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ABSTRACT 


The  primary  aim  of  this  study  has  been  the  development  and 
test  of  methods  for  evaluating  operator  performance  during  field 
test  of  complex  man-machine  systems.  Recommendations  are  presented 
whicn  are  aimed  at  strengthening  the  human  factors  evaluation 
process  in  both  the  field  evaluation  and  the  development  phases 
of  a  system.  However,  during  the  study  field  evaluation  took 
precedence  with  respect  to  development  and  trial  of  test 
procedures  and  techniques. 

Recommendations  are  made  based  upon  the  study  of  the 
development  and  field  test  of  specific  Naval  aircraft  systems. 

Field  evaluation  techniques  and  methods  were  tested  during 
the  evaluation  of  one  of  these  systems  -  the  P-JC  anti-submarine 
warfare  system.  These  recommendations  deal  with  the  identification 
of  measurement  noints,  measurement  scales,  evaluation  criteria  and 
levels  and  conditions  of  test.  The  assignment  and  training  of 
human  factors  test  and  evaluation  personnel  is  also  considered. 
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1.0  INTRODUCTION 


l.l  OBJECTIVE  AND  CONDUCT  OF  THE  STUDY 

The  objective  of  this  effort  has  been  the  development  of  more 
comprehensive  and  reliable  means  for  carrying  out  Human  Factors 
evaluation  during  field  testing  of  complex  man-machine  systems. 

Its  purpose  has  been  to  provide  recommendations,  procedures  and 
methods  useful  for  meeting  the  requirement  for  test  and  evaluation 
of  military  systems  before  acceptance  for  production  and  operational 
use. 


While  major  emphasis  has  been  placed  upon  field  test  and 
evaluation  just  prior  to  final  acceptance  of  the  system,  the 
contribution  and  importance  of  test  and  evaluation  during  the 
entire  development  cycle  is  considered  in  some  detail  in  this 
report. 

In  carrying  out  the  project  the  Research,  Development,  Test 
and  Evaluation  CRDT&E)  process  within  the  Navy  was  examined  in 
detail.  After  examining  a  number  of  different  types  of  systems 
undergoing  test  and  evaluation,  specific  aircraft  systems  were 
selected  for  detailed  study  in  the  development  and  evaluation  of 
techniques  and  methods  of  Human  Factors  field  testing.  These 
were  the  A-7A  and  the  P-3C  ASW  system.  The  major  portion  of  the 
work  was  carried  out  using  the  P-3C. 

In  brief,  the  A-7A  task  analyses,  mock-up  inspection  reports 
and  field  evaluation  efforts  were  Investigated  in  detail  after  the 
Principal  Investigator  had  participated  in  the  manufacturer's 
Indoctrination  course  on  the  system.  However,  the  major  effort  in 
testing  various  methods  and  techniques  was  conducted  using  the  P-3C 
system.  Both  an  Operational  Sequence  Diagram  (OSD)  and  a  Mission 
Time  Line  (MTL)  were  developed  for  the  Tactical  Coordinator's 
station  within  the  context  of  a  standard  Evaluation  Mission.  Using 
both  the  OSD  and  the  MTL  several  groups  of  experienced  Tactical 
Coordinators  (TACCO's)  participated  in  tests  designed  to  determine 
the  feasibility  of  Identifying  operator  performance  measurement 
points  within  the  system. 

The  MTL  was  used  to  test  the  feasibility  of  obtaining  estimates 
of  TACCO  workload  in  the  early  stages  of  work  with  the  P-3C. 

Although  the  feasibility  of  the  method  was  demonstrated,  the 
procedure  is  not  discussed  in  detail  in  this  report  since  the 
estimate  of  workload  from  the  MTL  is  more  properly  a  part  of  the 
system  development  phase. 


Tests  of  segments  of  the  mission  were  conducted  In  the  P-3C  weapon 
system  trainer.  Naval  Training  Device  Center  device  2F87,  using 
experienced  TACCO's  as  operators  to  determine  the  feasibility  of  its 
use  at  the  field  test  level. 

The  detailed  study  of  specific  aircraft  systems  was  supplemented 
by  an  extensive  review  of  formal  and  Informal  reports  and  writinqs 
relevant  to  the  problem  and  by  interviews  with  human  factors  personnel, 
administrators  and  evaluation  project  personnel.  A  reference  list  of 
the  reports  found  most  helpful  are  given  in  the  Bibliography  and  Source 
Material  section  of  this  report. 

1.2  ORGANIZATION  OF  THE  REPORT 

This  report  considers  the  Importance  of  continuous  evaluation 
throughout  the  development  phase  of  the  system  but  concentrates  upon 
the  problem  of  test  and  evaluation  just  prior  to  system  acceptance. 

The  role  of  various  evaluative  methods  applicable  throughout  the 
development  and  test  of  the  system  are  discussed  in  the  early  sections 
of  the  report  along  with  a  frame  of  reference  with  which  to  approach 
the  problem  of  test  and  evaluation.  Some  Important  personnel  and 
organizational  considerations  are  then  taken  up  before  the  suggestions 
and  recommendations  for  field  evaluation  proper  are  discussed. 

In  the  section  dealing  with  field  evaluation  processes  and 
recommendations  the  Importance  of  setting  forth  a  standard  Evaluation 
Mission  Is  stated.  The  use  of  the  Operational  Sequence  Diagram  for 
identification  of  measurement  points,  both  for  manual  performance  and 
decisions  is  described  and  is  followed  by  discussion  of  means  of 
recording  and  evaluation  Including  the  problem  of  establishing  criterion 
performance.  The  several  test  conditions  under  which  meaningful 
evaluative  measures  may  be  obtained  are  discussed  in  order  that  the 
Human  Factors  evaluator  may  consider  test  situations  other  than  the 
actual  system  as  sources  of  useful  and  predictive  evaluative  data. 
Finally,  the  subjects  of  workload  analysis,  system  evaluation  for 
operator  feedback  and  the  use  by  the  operator  of  old  procedures  and 
habits  are  discussed. 
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2.0  A  HUMAN  FACTORS  EVALUATION  FRAME  OF  REFERENCE 


A 


2.1  TOTAL  SYSTEM  VS.  COMPONENT  EVALUATION 

In  considerinq  tne  test  and  evaluation  of  a  particular  system  it 
might  be  assumed  that  the  tota I  system  is  in  some  way  evaluated  with 
resoect  to  its  capabilities  for  achievinq  some  specified  mission. 

Under  this  assumption  it  miqht  be  concluded  that  the  field  test  of 
the  system  is  a  simple  matter  of  determining  on  a  "  go  no-qo"  basis 
whether  or  not  the  system  meets  the  mission  criteria.  If  such  were 
the  case,  specific  concern  with  evaluation  of  the  human  component  or 
any  other  component  of  the  system  would  seem  to  be  unnecessary  wnen 
carried  out  in  connection  witn  the  test  of  the  system. 

The  assumption  that  human  factors  evaluation  Is  a  necessary  and 
somehow  distinctive  part  of  tne  evaluation  Drocess  requires  some 
justification  and  clarification.  An  answer  seems  to  be  needed  to  the 
question  as  to  why  there  should  be  concern  with  the  minutiae  of 
evaluatinq  components  and  subsystems  of  a  system  if  the  evaluative 
decision  is  one  of  "acceot"  or  "reject"  the  total  system.  Presumably 
if  tne  total  system  successfully  accomplishes  its  desinn  mission  there 
would  be  no  interest  in  measuring  the  performance  of  any  of  its 
components  -  human  or  hardware.  If  the  system  meets  its  mission 
criteria  it  may  be  assumed  that  the  hardware  and  human  components  are 
functioning  so  as  to  bring  about  total  mission  success. 

There  are,  in  fact,  practical  and  coqent  reasons  why  component 
and  subsystem  evaluation  is  necessary.  The  reasons  for  conducting 
component  and  subsystem  evaluation  rather  than  an  overall  sys+em 
evaluation,  when  examined  closely,  tend  to  brina  into  focus  the  re¬ 
quirements  for  carrying  out  an  adequate  human  component  field  evaluation. 
The  following  paragraphs  are  intended  to  clarify  the  problem  and  form 
a  rationale  for  the  recommendations  given  later. 

2.2  THE  NATURE  OF  THE  DEVELOPMENT  PROCESS 

Tne  first  important  reason  for  being  concerned  with  component 
testing  during  system  evaluation  stems  from  the  nature  of  the  development 
process  Itself.  In  theory,  at  tne  beginning  of  the  development 
cycle  the  mission  of  the  system  is  delineated  in  detail  wltn  criteria 
for  successful  performance  clearly  spelled  out.  Many  considerations 
mitigate  against  sucn  a  clear  delineation. 

When  a  system  is  developed  for  a  new  mission  or  to  extend  the 
capabilities  for  executinq  a  present  mission,  the  details  of  the 
system  and  its  performance  criteria  cannot  be  stated  definitively 
at  the  outset.  Generally,  rather  explicit  overall  system  criteria 
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are  established  to  be  attained  through  application  of  the  present 
state  of  the  equipment  art  or  tne  projected  state  of  that  art. 

However,  details  of  the  mission  and  the  performance  criteria  develop 
with  the  development  of  the  system  or  more  precisely,  with  the 
development  of  the  hardware  for  the  system.  As  the  iterative 
process  jf  design  proceeds  the  details  of  system  components  and 
their  requisite  individual  performance  criteria  emerge.  The  overall 
goal  or  mission  of  the  system  is  broken  down  into  intermediate  or 
secondary  goals  for  subsystems  and  components  of  the  total  system. 

The  attainment  of  these  intermediate  or  secondary  goals  by  the 
components  and  subsystems  are  intended  to  cascade  summating  in 
attainment  of  the  overall  system  goal. 

During  the  development  process  a  system  Is  being  synthesized 
from  components  chosen  after  an  analytical  exercise  In  which  the 
total  system  requirements  have  been  broken  down  InTO  subsystem  and 
component  requirements.  Actually  the  processes  of  analysis  ar.d 
synthesis  go  on  In  iterative  fashion  throughout  the  development 
period.  The  central  point,  however,  is  that  svstem  synthesis  is 
attained  through  selection  of  components  and  subsystems  which  can 
perform  in  accordance  with  the  requirements  made  explicit  by  the 
analysis.  Components  are  chosen  (I)  whose  input-output  character¬ 
istics  match  adjacent  components,  (2)  which  oerform  the  proper 
transformations  on  The  input  and  (3)  which  perform  their  proper 
function  within  the  required  time.  Component  and  subsystem 
performance  summate  to  total  system  performance.  One  needs  only 
to  reflect  on  the  process  of  synthesizing  a  simple  audio  circuit  to 
understand  how  the  incompatibility  of  one  component  can  lead  to 
total  system  failure. 

Field  evaluation  of  a  system  takes  its  cue  from  the  development 
philosophy.  For  most  systems  total  system  effectiveness  in  field 
evaluation  is  an  estimate  based  upon  evaluations  of  components  and 
subsystems  of  the  system.  Most  field  evaluations,  therefore,  are  not, 
and  probably  cannot  be  for  systems  r-f  any  complexity,  evaluations  of 
tne  system  in  toto  in  its  intended  operational  environment.  Rather 
tney  are  evaluations  of  particular  components  and  subsystems  as 
they  operate,  in  combination  with  other  parts  of  the  system  in  an 
environment  more  or  less  representative  of  the  intended  operational 
environment.  The  tested  components  (both  hardware  and  human)  are 
intended  to  be  representati ve  of  those  which  will  finally  comprise 
the  total  system.  These  intentions  are  often  only  approximately 
realized.  In  field  evaluation  there  is  the  need  to  choose  the  proper 
components  and/or  subsystems  for  test  and  the  representative 
environmental  conditions  under  which  to  test  them  If  the  testing 
is  to  provide  an  accurate  estimate  of  how  well  the  total  system  will 
function  In  its  operating  environment. 


Since  system  development  is  a  process  of  synthesizing  components 
and  evaluation  is  the  testing  of  those  components  emphasis  must  be 
given  to  the  point  made  by  other  writers  and  which  will  be  discussed 
further  in  this  report.  That  is,  in  order  to  properly  evaluate  a 
component  of  the  total  system  it  is  necessary  that  the  evaluator  know 
explicitly  the  role  of  that  component  in  the  total  system.  The  human 
operator  Is  such  a  component.  The  Human  Factors  evaluator  must  know 
what  the  operator  must  do,  how  wel I  he  must  do  it  and  under  what 
environmental  conditions  he  must  perform.  This  determination  cannot 
be  left  to  last  minute  speculations  by  the  evaluator  in  the  field. 

2.3  THE  TROUbLE-SHOOT I NG  ASPECT  Of  EVALUAT ION 

A  second  reason  and  necessity  for  evaluation  at  the  subsystem 
and  component  level  comes  about  when  a  particular  chain  of  components 
or  subsystems  is  tested  and  the  performance  fails  to  meet  the  standard. 
Under  these  circumstances  it  is  necessary  to  determine  which  sub- 
system(s)  or  component(s)  failed  to  perform  to  their  particular  criteria. 
The  Human  Factors  evaluator  is  interested  in  determining  whether  the 
human  component  failed  and,  if  so,  in  what  way. 

These  circumstances  require  that  information  on  performance  of 
subsystems  or  components  be  obtained  in  order  to  diagnose  the  source 
of  difficulty.  This  information  must  be  obtained  through  a  systematic 
and  reliable  means  suitable  for  identifying  the  trouble  spot  within 
the  larger  unit  after  the  larger  unit  has  failed. 

2.4  the  adaptive  human  component 

A  tnird  reason  for  component  evaluation  is  peculiar  only  to  the 
human  component.  It  is  not  unusual  that  the  ingenuity  and  adapta¬ 
bility  of  the  human  operator  enables  him  to  perform  in  a  way  whicn 
results  in  system  success  even  though  his  actions  and  performance  may 
have  been  quite  different  from  those  anticipated  by  the  designer.  It 
is  this  adaptability  and  ability  to  recover  which  characterizes  the 
human  component  of  the  system,  makes  human  factors  evaluation  (or 
data  collection)  important  in  all  systems,  and  which  differentiates 
the  human  component  evaluation  from  that  of  other  components  of  the 
system.  A  method  for  determining  when  and  how  the  operator  has 
performed  in  this  adaptive  way  is  necessary  for  guiding  redesign, 
procedural  changes  or  trainina. 


3.0  LEVELS  OF  EVALUATION 


3.1  THE  ROLE  AND  DEFINITION  OF  FIELD  EVALUATION 

In  workina  with  the  problem  of  Human  Factors  evaluation  restricting 
the  study  to  "field"  evaluation  creates  a  certain  dilemma  since  it  is 
difficult  to  define  where  such  evaluation  beqlns  or  ends.  Althouqh 
there  are  formally  defined  evaluation  phases  such  as  the  Navy  Board  of 
InsDectlon  and  Survey  Trials  (BIS),  evaluation  is  a  continuous  process 
beginning  In  the  early  staqes  of  system  development.  Thus  the  field 
evaluation  should  grow  out  of  and  be  dependent  upon  a  pody  of  infor¬ 
mation  and  testing  of  the  system  during  Its  development  life.  It  may 
extend  well  into  the  operational  use  of  the  system. 

Further,  it  is  necessary  to  define  what  the  field  evaluation  is 
to  accomplish.  Broadly  speak inq  it  is  meant  to  test  the  system  aqainst 
assumptions  made  about  its  performance  as  it  was  originally  conceived. 

In  a  very  real  sense  any  test  which  predicts  how  well  the  system  will 
perform  is  desirable  at  whatever  point  in  the  development  process  it 
is  conducted.  The  element  which  is  added  to  test  and  evaluation  throuqh 
conduct  in  tne  field  is  presumably  that  variables  and  conditions  are 
more  nearly  representative  of  those  in  fleet  operations  and  therefore 
more  valid.  This  representativeness  varies  from  system  to  system 
depending  upon  such  factors  as  similarity  of  the  system  to  previous 
systems,  urgency  of  system  need  and  availability  of  test  personnel  and 
equipment. 

Tests  conducted  toward  the  end  of  the  development  cycle  generally 
will  nave  greater  "content"  validity  than  those  conducted  early.  That 
is  to  say  that  tne  equipment  to  be  tested  and  the  conditions  of  test 
will  be  judged  by  competent  evaluators  to  be  good  likenesses  of  the 
ultimate  criteria,  i.e.,  operational  use.  However,  deficiencies  found 
in  the  system  at  this  staqe  are  generally  more  costly  and  difficult  to 
correct  than  if  found  earlier  in  the  development  cycle.  It  is  desirable 
therefore,  that  valid  evaluation  of  component  performance,  human  and 
hardware,  be  carried  out  as  early  in  the  development  process  as  possible 
The  more  valid  the  component  testina  early  in  development  the  fewer 
problems  that  will  arise  during  field  testing  or  in  subsequent 
operational  use.  The  most  desirable  situation  is  one  in  which  tests 
carried  out  at  all  stages  of  development  and  field  test  have  high 

validity  for  predicting  performance  in  operations.  This 
validity  in  which  empirical  relationships  Detween  tests 
and  operational  performance  are  established  cannot  be  obtained  until 
reliable  measures  of  performance  In  operations  are  possible  -  something 
toward  which  more  effort  should  be  directed. 

At  some  point  before  actual  operational  deployment  an  evaluation 
must  oe  made  to  determine  whether  the  system  performs  to  the  original 
conception  and  specifications.  This  is  the  role  generally  assigned 


predictive 
pred i ct ! ve 


to  f i g I o  test.  The  results  of  this  evaluation  can  and  should  also 
serve  as  criteria  against  which  to  validate  tests  conducted  earlier 
in  tne  develooment  process.  Valid  testing  conducted  earlier  in 
development  will  prevent  des i on  deficiencies  from  reachinq  the 
accept-reject  field  test  point  and  their  necessary  expensive  correction. 

3.2  VALIDATION  AND  USL  OF  EARLIER  TESTING 

It  nas  been  indicated  that  field  testing  must  serve  to  test  the 
assumptions  and  specifications  made  durinq  system  conception  and 
development  and  as  criteria  anainst  which  to  validate  testinn  carried 
out  earlier  in  the  development  phase.  These  earlier  tests  take  a 
variety  of  forms. 


3.2.1  Physical  Models 

Physical  models  of  the  system  ranqe  from  a  simple  static  mock-uo 
of  a  part  of  the  system  tnrouqn  a  dynamic  simulator  to  the  aircraft 
itself.  This  section  discusses  these  models  and  their  role  in  human 
factors  evaluation. 

3. 2.1. 1  Static  Mock-Up.  At  present  the  physical  model  most  used  in 
numan  factors  design  and  evaluation  is  the  operator  station  mock-up. 

Tnis  mock-up  is  used  almost  continuously  as  ar.  evaluative  tool  durinq 
the  design  process.  It  is  also  the  focal  point  durina  a  formal  human 
factors  evaluation,  i.e.,  the  mock-up  inspection.  It  is  believed  that 
major  improvements  can  be  made  in  the  procedures  and  techniques  used 
durinq  these  inspections  as  they  are  now  constituted.  This  belief  has 
ueen  found  to  be  almost  universally  supported  by  those  having  experience 
witn  the  mock-up  inspection. 

As  a  background  for  discussing  the  mock-up  inspection  and  tne  role 
it  plays  in  overall  evaluation  it  is  necessary  to  discuss  two  levels  of 
evaluation.  The  first  level  is  termed  "operator  station  oriented" 
evaluation.  The  second  is  termed  "mission-oriented". 


In  an  cperaror  station  oriented  evaluation  tho  mock-up  is  evaluated 
for  compatibility  witn  i he  operator's  capabilities  and  limitations. 

Fne  operator  station  is  examined  for  its  conformity  with  good  human 
engineering  principles  and  handbook  data.  Evaluation  is  made  on  the 
oasis  of  wcrk-place  layout,  control  coding,  contro I -d i sp I  ay  relation¬ 
ships,  illumination,  anthropometr i c  compatibility  and  the  like. 


To  carry  out  a  station  oriented  human  factors  evaluation  personnel 
must  be  knowledgeable  witn  respect  to  the  human  factors  literature 
and  data,  be  able  to  critically  evaluate  those  data,  and  extrapolate 
from  tnem  in  the  light  of  the  particular  svstem  being  evaluated.  Further, 
they  must  be  cognizant  of  tne  principles  of  good  human  factors  desian 
as  applied  to  tne  specific  system  being  evaluated.  The  points  to  be 
covered  during  the  evaluation  should  be  incorporated  into  a  checklist 
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to  insure  that  important  areas  are  not  overlooked.  The  Human  Factors 
evaluator  should  be  knowledgeable  with  respect  to  the  personal 
equipment  to  be  worn  by  the  operator  and  evaluations  should  be 
conducted  takinq  into  account  the  effect  of  personal  equipment  upon 
performance. 


Mission  oriented  evaluation  requires  an  examination  of  tne 
operator  station  in  the  light  of  operator  functions  and  tasks  and  an 
assessment  of  whether  the  system  will  perform  to  its  designed  level. 

In  order  to  carry  out  such  an  evaluation  it  is  necessary  to  know  i r. 
detail  what  functions  and  tasks  the  operator  is  required  to  perform. 

A  determination  must  De  made  as  to  whether  the  operator  can  carry  out 
his  functions  and  tasks  in  the  proper  sequence  to  the  required  accuracy 
within  the  required  time.  Therefore,  the  mission  oriented  evaluation 
requires  that  the  evaluator  have  detailed  information,  by  mission 
segment,  about  subsystem  functions,  data  flow,  and  the  requirements 
placed  upon  the  operator.  This  information  may  be  oDtained  tnrougn 
the  detailed  Mission  Time  Line  (MTL)  or  Operational  Sequence  Diagram 
(OSD)  of  the  system.  These  are  discussed  in  more  detail  in  Section 
5.0  in  connection  with  the  problem  of  establishing  performance 
measurement  points. 


3.2. 1.2  Simulators  -  The  simulator,  as  a  dynamic  physical  model  of 
the  crew  station,  has  several  advantages  over  tne  static  mock-up. 

From  tne  evaluator's  point  of  view  it  offers  flexibility,  opportunity 
to  obtain  reliable  performance  data  and  a  test  situation  more 
representative  of  the  real  system.  It  allows  for  the  evaluation  of 
the  dynamic  man-machine  interactive  performance. 

Advances  in  simulator  design  and  construction  are  steadily 
increasing  its  utility  in  system  desiqn  and  evaluation,  Wnen  sucn  a 
device  can  be  made  available  it  is  much  to  be  preferred  over  the 
static  mock-up  for  use  durinq  Mock-Up  Inspection.  The  use  of  tne 
weapon  system  simulator  to  the  qreatest  extent  possible  for  performance 
testing  throughout  the  development  cycle  and  during  field  evaluation 
is  to  Pe  recommended.  More  is  said  apout  iis  use  durinq  field  testing 
in  Section  5.7  of  tnis  report. 

3.2.2  Part  Task  Testing 


Active  concern  witn  numan  factors  evaluation  from  tne  beginning 
of  tne  system  allows  for  tne  introduction  of  a  number  of  evaluative 
tecnniaues.  Particularly  useful  is  tne  tecnnigue  wnicn  nas  been 
termed  tne  "open-loop"  test  of  display  design  configurations  and 
whicn  may  be  used  in  weeding  out  or  narrowing  down  design  alternatives 
in  the  early  stages  of  development.  These  techniques  are  essentially 
tach i stoscop i c  presentations  of  display  designs  and  are  used  for 
comparative  evaluation.  They  are,  in  fact,  the  techniques  through 
which  a  major  portion  of  the  human  factors  data  available  to  us  today 
was  obtained. 


This  testinq  technique  is  termed  "open- loop"  since  the  response 
of  the  suoject  has  no  direct  effect  upon  the  next  stimulus  presentation. 
The  stimulus  material  may  De  a  display  configuration  presenting  infor¬ 
mation  which  the  suDject  is  required  to  interpret,  readout,  and  reDort. 
Measures  of  tne  subjects  performance  may  be  the  speed,  accuracy  or 
ooth  with  which  the  information  is  read  out.  Th9  stimulus  material 
used  may  be  in  tne  form  of  a  static  projection,  i.e.,  a  3b  mm  slide* 
in  such  case  it  is  termed  static  open-loop  testing.  Alternatively, 
the  material  may  be  presented  through  use  of  motion  picture  film  in 
which  display  elements  move  realistically.  This  type  of  testing  is 
then  termed  "dynamic  open-loop"  testing.  A  detailed  description  of 
this  type  of  testinq  is  given  in  Schum,  Elam  and  Matheny  (1962)  in 
which  its  use  has  been  demonstrated. 

3.3  PHASING  uF  EVALUATION  LEVELS 

The  initial  point  to  De  made  is  that  human  factors  evaluation 
should  begin  at  the  point  of  assiqnment  of  functions  in  the  development 
process  and  continue  through  the  use  in  operations.  During  this 
evaluation  process  at  least  five  methods  of  evaluation  are  applicable. 
Tnese  are  (I)  numan  engineering  check-lists,  (2)  static  and  dynamic 
open-loop  tests,  (3)  mock-up  evaluations,  (4)  tests  in  the  simulator 
of  the  system  and  (5)  tests  In  the  actual  system.  A  suggested 
relationship  of  these  types  of  testinq  throughout  the  development 
and  employment  of  a  system  is  given  in  Figure  I. 

Human  engineering  check-lists  are  particularly  appropriate  to 
early  stages  of  human  engineering  evaluation  and  can  be  used  with  the 
mock-up  for  type  one  evaluations.  A  listing  of  check- lists  felt  to 
be  representative  of  those  in  use  is  given  in  the  Bibliography  and 
Source  Material.  Open- loop  testinq  lends  itself  to  comparative 
evaluations  of  information  display  components  and  subsystems.  The 
simulator  and  aircraft  are  most  effective  in  evaluating  whether  or 
not  human  performance  meets  specific  criteria  for  system  effectiveness. 

Under  this  conceptualization  the  mock-up  inspection  as  such  takes 
on  a  different  meanlnq.  Under  it  the  mocx-up  of  the  operator  station 
as  a  desiqn  and  evaluation  tool  is  used  on  a  conti nuina  basis  by 
manufacturer  personnel  under  continuina  monitorship  of  user  personnel. 
The  si qni fiance  of  and  necessity  for  formal  mock-up  inspections  are 
considerably  reduced. 
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Extent  of  use 


Human  Enqineerinq 
Check- I ists 


1.  Human  enqineerinq  check- I ists  used  to  evaluate  deslqn  compatibility 
with  numan  capabilities  and  limitations. 

2.  Open- loop  tests  used  for  comparative  evaluations  of  display  and  con¬ 
figuration  desiqn. 

3.  Mock-up  used  in  conjunction  with  human  englneerinq  check-lists  to 
determine  design  compatibility  with  human  capabilities  and  limitations. 

4.  Simulator  used  to  evaluate  operator  performance  against  performance 
criteria.  In  operational  use  it  may  be  used  in: 

o  Evaluating  proposed  new  or  modified  tactics 
o  Accident  investigation 

o  Diagnosis  of  performance  which  Is  below  criterion  requirements 

5.  Actual  system  used  to  evaluate  aqainst  performance  criteria  and  to 
evaluate  proposed  new  or  modified  tactics. 


*  PE  -  Preliminary  Evaluation 


Fiqure  I.  Relationship  and  extent  of  use  of  test 
procedures  during  system  development  and  evaluation. 
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4.0  PERSONNEL  ANU  ORGANIZATIONAL  CONSIDERATIONS 


4.1  DOCUMENTATION 

In  the  recent  past  the  requirements  for  consideration  of  human 
factors  in  the  desiqn  and  evaluation  of  military  systems  has  become 
more  explicitly  documented.  This  comes  about  throuqh  adoption  of 
Ml L-H-46855,  Human  Engineering  Requirements  for  Military  Systems, 
Equipment  and  Facilities,  16  February  1968  and  Ml L-STD- 1 472,  Human 
Engineering  Design  Criteria  for  Military  Systems,  Equipment  and 
Facilities,  9  February  1968.  These  two  documents  when  cited  in  a 
contract  specification  provide  authority  for  carryinq  out  effective 
human  factors  effort.  However,  the  documents  themselves  cannot 
spell  out  the  requirements  in  the  deTail  which  ensures  accomplishment 
of  a  good  human  factors  effort  in  tne  absence  of  trained  personnel 
in  sufficient  numbers  dedicated  to  makinq  their  contribution  felt. 

4.2  MANAGEMENT 

For  human  factors  evaluation,  assignment  of  tne  right  personnel 
oegins  at  the  project  office  level  with  responsibility  for  the 
evaluation  function  vested  in  a  designated  individual.  It  should  be 
his  responsibility  to  see  that  the  system  and  time  line  analyses 
contain  the  information  on  test  points  and  their  priorities  (see 
Section  5.0).  These  test  points  should  be  within  the  context  of  an 
evaluation  mission.  This  mission,  set  forth  early  in  development, 
will  evolve  in  detail  as  the  equipment  configuration  becomes  firmed 
up.  It  is  the  framework  upon  which  both  Human  Engineering  design  and 
evaluation  will  hang.  The  human  factors  engineer  in  the  Project  Office 
must  insure  that  this  mission  and  the  evaluation  test  points  are 
developed. 

4.3  ASSIGNMENT  OF  CUSTOMER  PERSONNEL  TO  CONTRACTOR  FACILITIES 


The  assignment  of  customer  personnel  to  the  human  factors  effort 
at  the  contractor's  facility  is  highly  recommended.  It  is  suggested 
that,  optimally,  these  personnel  be  graduates  of  specialist  schools 
witn  special  additional  training  in  human  factors  and  recent  experience 
in  systems  similar  to  the  one  beina  evaluated.  The  importance  of  the 
human  component  to  the  system  and  the  scarcity  of  hard  data  on  man's 
performance  in  such  systems  warrants  giving  especial  attention  to  the 
qualifications  of  the  desiqn  and  evaluation  personnel  who  deal  with 
him. 
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r  personnel  should  be  selected  for  their  interest 
qualifications  for  human  factors  work.  Analogous 
test  pilots  having  the  additional  qualifications 
other  engineering  degrees  the  human  factors 
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specialty  should  require  special  qualifications  In  human  engineering. 

A  special  course  of  three  to  four  weeks  minimum  shout'd  be  made 
available  for  this  specialty  coming  after  the  individual  has 
qualified  as  a  specialist,  e.g.,  a  Navy  Test  Pilot.  This  special 
training  course  is  discussed  further  In  Section  4.5  and  a  suggested 
course  outline  is  given  in  Appendix  A.  This  course  is  suggested  as 
a  minimum  requisite  and  not  a  substitute  for  formal  training  In  Human 
Factors.  The  responsibilities  of  these  personnel  at  the  contractor’s 
facility  would  be  purely  advisory  In  helping  to  develop  tne  evaluation 
mission,  critical  test  points  and  methods  of  measurement.  They  would 
become  members  of  the  evaluation  team  during  mock-up  Inspection  lending 
their  detailed  knowledge  of  the  operator  reguirements  and  the  data  flow 
through  the  system  to  a  more  objective  and  mission  oriented  evaluation. 

4.4  CONTRACTOR  ASSIST  DURING  CUSTOMER  EVALUATIONS 

During  evaluations  carried  out  at  the  customer's  facilities  it  is 
recommended  that  contractor  human  factors  personnel  assist  and  advise 
on-site  in  the  planning  and  conduct  of  the  tests.  These  contractor 
personnel  would  contribute  their  knowledge  of  the  system  to  the  planning 
and  conduct  of  the  tests.  Their  detailed  knowledge  will  help  the 
customer  evaluator  immeasureably  in  working  out  the  details  of  how  to 
determine  test  points  and  in  diagnosing  sources  of  operator  difficulties. 

4.5  TRAINING  OF  EVALUATION  PERSONNEL 

The  approach  to  be  taken  by  the  evaluator  in  any  evaluation  is 
essentially  that  which  the  serious  experimenter  would  take  in  testing 
an  hypothesis.  He  must  be  as  knowledgeable  and  have  as  much  quantitative 
information  as  possible  about  the  variables  and  conditions  influencing 
the  operator  behavior.  He  must  either  control  these  or  be  able  to 
assess  their  effects.  He  must  also  have  an  understanding  of  experimental 
design,  of  reliability  of  measurement  and  of  data  analysis  and  report. 

An  appreciation  of  these  requirements  coupled  with  experience  and 
knowledge  of  the  operational  conditions  under  which  the  system  will 
function  would  combine  to  maximize  the  effectiveness  of  the  human 
factors  evaluator.  Their  combination  in  a  single  individual  is  rare. 

A  training  program  designed  to  produce  such  a  combination  is  recommended. 

The  human  factors  enqineer  usually  comes  to  the  evaluation 
situation  with  a  limited  knowledoe  and  appreciation  of  the  operational 
demands.  The  customer  project  personnel  assigned  to  evaluate  the 
system  have  the  operational  experience  but  usually  are  not  experienced 
in  the  methods  of  experimentation  which  should  be  applied.  A  cross 
training  program  is  recommended. 

It  us  suggested  that  customer  project  personnel  could  become 
oriented  and  minimally  knowledgeable  about  experimental  methods 
throuah  an  indoctrination  course  of  three  to  four  weeks  minimum. 

This  course  would  be  offered  to  project  officers  and  project  pilots 
who  are  directly  concerned  with  planning  and  conducting  the  evaluation. 
The  course  outlined  in  Appendix  A  is  suggested.  After  such  a  course 
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a  person  could  not  be  considered  an  authority  in  all  areas  of 
evaluation.  Rather,  he  would  gain  an  overall  perspective  of  the 
requisites  for  evaluation  and  appreciate  the  need  for  consultation 
and  assistance  from  subject  matter  specialists  in  such  areas  as 
experimental  design,  performance  measurement  and  rating,  and 
analysis  of  data. 

Human  factors  personnel  should  be  indoctrinated  in  the 
operational  use  of  like  systems  in  every  way  possible.  For  example, 
within  the  Navy  it  has  been  suggested  that  short  tours  aboard 
carriers  by  human  factors  personnel  should  be  undertaken.  Every 
opportunity  for  these  personnel  to  observe  the  operation  of  similar 
systems  either  in  the  operational  theater  or  in  training  operations 
should  be  taken. 
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5.0  FIELD  EVALUATION  PROCESS  AND  RECOMMENDATIONS 


5.1  OUTLINE  OF  THE  FIELD  EVALUATION  PROCESS 

As  indicated  in  earlier  sections  of  this  report  the  field 
evaluation  is  mission  oriented  and  Is  designed  to  assess  whether 
the  system  will  perform  to  some  specified  criterion  level.  In 
order  to  carry  out  such  an  evaluation  for  the  human  component 
it  Is  necessary  to  know  In  detail  what  tasks  the  operator  must 
perform,  In  what  sequence,  and  to  what  criteria.  It  is  also 
necessary  that  +he  operating  environment  of  the  system  be  thoroughly 
understood  so  that  the  important  aspects  of  that  environment  may 
be  incorporated  into  the  test  and  evaluation  process.  A  deter¬ 
mination  must  be  made  as  to  whether  the  operator  can  carry  out  his 
functions  and  tasks  In  the  proper  sequence  to  the  required  accuracy 
within  the  required  time  under  conditions  representative  of  the 
operational  envi ronment. 

The  suggested  steps  for  carry inq  out  the  Human  Factors  field 
evaluation  are  given  In  Figure  2.  A  brief  discussion  of  the  items 
shown  In  this  figure  is  given  as  an  orientation  to  the  overall  human 
factors  test  and  evaluation  process. 

Normally  Steps  I  through  4  and  8,  9  and  II  will  have  been 
accomplished  during  system  development  as  basic  information  for 
design  and  evaluation  and  will  be  available  as  data  for  planning 
the  field  evaluation.  If  they  have  not  it  will  be  necessary  that 
the  best  approximation  to  them  possible  be  carried  out  early  In  the 
field  evaluation  phase. 

The  Evaluation  Mission  of  Step  I  in  Figure  2  Is  discussed  in 
detail  in  Section  5.3.  This  mission  is  oriented  toward  the  tactical 
use  of  the  system  and  must  incorporate  those  design  elements  involved 
in  its  tactical  employment. 

The  sequent  la  I  task  listing  of  Step  2  is  derived  by  detailing 
the  actions  of  the  operator  within  the  seaments  of  the  evaluation 
mission.  This  step  is  detailed  In  Section  5.4, 


In  Step  3  the  sequential 
Sequence  Diagram  (OSD)  format. 
The  OSD  will  be  of  primary  use 
been  omitted,  whether  they  are 
recommended  to  be  used  I n  Step 
measurement  points. 


task  list  is  placed  in  the  Operational 
This  is  described  in  Section  5.4. 
in  determining  whether  tasks  have 
in  the  proper  sequence,  and  is 
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Recommended  procedure  for  measuring  and  evaluation  operator  performance 


In  Step  4  the  task  sequence  derived  from  the  OSD  is  transferred 
to  a  Mission  Time  Line  (MTL)  format.  The  MTL  provides  a  check  on 
the  proper  sequencing  of  operator  tasks  and  provides  the  basic 
information  for  deriving  the  workload  values  of  Step  II. 

When  computer  equipment  and  facilities  are  available.  Steps  5, 

6,  and  7  are  highly  recommended.  This  process  is  discussed  In 
Section  5.4.  Computer  I z i ng  the  generation  of  the  OSD  and  the  MTL 
enables  the  evaluator  to  incorporate  most  quickly  the  changes  that 
occur  in  the  system  equipment  and  task  sequence  during  evaluation. 

In  Step  10  of  Figure  2  the  test  and  evaluation  procedures  for 
measurinq  operator  performance  are  detailed. 

The  Operational  Sequence  Diaqram  Is  used  for  identifying  points 
within  the  system  at  which  the  operator  and  his  equipment  interact 
and  at  which  performance  may  be  measured.  The  characteristics  of 
each  of  the  points  selected  will  govern  the  type  of  data  to  be 
collected  and  will  provide  insight  and  information  as  to  how  the  data 
may  be  recorded  and  analyzed.  This  process  is  covered  in  Section  5.6. 
Methods  of  collecting  and  recording  data  at  the  measurement  points 
which  have  been  identified  are  also  discussed  ir.  that  Section, 

The  proplem  of  determining  tne  evaluation  criteria  for  each  of 
the  measurement  points  selected  is  discussed  in  Section  5.6.3.  The 
measures  taken  in  Step  14  are  evaluated  using  the  specified  criteria 
for  successful  performance.  The  steps  to  be  taken  when  no  criteria 
are  available  are  also  discussed.  This  evaluation  leads  to  either 
system  acceptance.  Step  17,  or  system  modi fi f icat ion,  Step  18.  Where 
system  mod i f i f icat ion  occurs  the  changes  in  the  evaluation  mission 
and  the  operator's  tasks  are  Incorporated  into  the  OSD  and  MTL  and 
the  appropriate  re-evaluation  of  the  operator  performance  made. 

5.2  SOURCES  OF  INFORMATION 

In  order  to  evaluate  the  performance  of  the  human  operator  during 
field  evaluation  of  a  system,  it  is  important  that  certain  information 
be  available  describing  the  requirements  placed  on  the  operator  by 
the  specific  equipment  being  evaluated.  The  major  portion  of  this 
information  normally  would  be  generated  during  weapon  system 
development  since  this  process  requires  a  knowledge  of  or  projections 
about  (I)  mission  requirements,  (2)  equipment  capability,  (3)  vehicle 
performance  and  limitations,  and  (4)  crew  capabilities  and  limitations. 
However,  continuous  updating  of  the  I ntormar i on  is  necessary  through 
the  test  and  evaluation  phase  as  the  components  or  subsystems  of  the 
system  may  be  changed,  equipment  may  not  perform  to  expectations, 
and  tactics  and  procedures  may  be  modified. 
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The  Human  Factors  evaluator  must  be  well  informed  about  the 
system  requi rements  and  system  desiqn.  Only  throuqh  a  thorouqh 
Knowledge  of  the  system  can  he  create  or  update  the  task  analyses. 

His  work  will  be  affected  continuously  by  chanqes  in  system  desiqn 
and  equipment  capability.  He  must  be  equipped  to  incorporate  tnese 
chanqes  into  his  analysis  readily  and  modify  his  test  plan 
accordingly.  If  the  evaluator  has  worked  closely  with  the  desiqn 
and  test  of  the  system  throuqh  its  development,  ho  will  be  well 
prepared  for  the  field  test  phase  in  respect  to  knowledqe  of  the 
system.  Normally  he  will  not  have  this  experience. 

The  methods  Dy  which  information  about  the  system  can  be 
obtained  by  the  evaluator  are  (I)  study  of  relevant  documents, 

(2)  interviews  with  contractor  desiqn  personnel  and  (3)  inter¬ 
viewing  of  experienced  operators.  A  backqround  of  experience  with 
previous  versions  of  the  system  or  with  earlier  systems  for 
accomp ! i sh i nq  the  mission  will  be  extremely  beneficial  in  under- 
standinq  and  evaluatim  the  system.  The  relevant  documents  include 
but  are  not  limited  to  the  system  and  component  specifications, 
basic  mission  manuals  and  contractor  publications.  System  desiqn 
information  may  De  obtained  from  contractor  documents  and  throuqh 
interviews  with  contractor  personnel.  A  knowledqe  of  tne  operational 
environment  of  the  system  may  be  obtained  from  the  requirements 
documents,  tnrouqh  interviews  witn  operators  experienced  in  similar 
systems  ana  from  contractor  publications  reciting  and  assumptions 
made  during  desiqn  about  tne  operational  tactics  and  environment. 

3.3  THt  tV ALU AT  ION  MISSION 

Tne  purpose  of  tne  evaluation  mission  is  to  provide  a  standard 
test  situation  whicn  i ncorporates  as  many  of  tne  desiqn  elements  of 
tne  system  as  possible  and  which  represents  the  operational  system 
witn  respect  to  tactics  and  environment.  For  systems  in  wnicn  several 
unique  missions  may  be  accomp I isned  multiple  evaluation  missions 
will  oe  necessary.  If  a  detailed  deslan  mission  has  been  developed 
durinq  the  system  development  phase  tne  evaluation  mission  should 
parallel  it  closely.  The  evaluation  mission  may  need  to  be  varied 
from  the  desiqn  mission  because  of  available  test  facilities  ana  time 
and  budget  limitations. 

For  most  systems  being  evaluated  emphasis  snould  be  placed 
upon  the  tactical  phase  of  the  mission  since  it  is  in  this  phase 
tha*  goals  can  be  pin-pointed  and  the  success  or  failure  of  the 
system  is  determined.  For  example,  in  the  F-3U  system  the  tasks 
involved  in  taxi,  take-off,  climb-out,  cruise,  let-down  and  landinq 
are  necessary  to  accomplishment  of  tne  mission.  However,  for  tnis 
system  emphasis  would  first  be  placed  upon  evaluation  during  fhe 
tactical  phases  of  the  mission  in  whicn  the  ASW  equipment  is  being 
exercised. 
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Generally,  the  evaluation  mission  profile  will  begin  with  entry 
into  the  tactical  phase.  All  of  the  factors  that  affect  the 
successful  accomplishment  of  the  mission  must  be  considered.  If 
possible  the  evaluator  should  discuss  the  evaluation  mission  design 
with  operator  personnel  experienced  in  missions  similar  to  the  one  in 
question  to  insure  that  it  Is  feasible  and  representative  of  the 
operational  requirement.  The  tactical  phase  of  the  mission  should  be 
subdivided  Into  segments  and  may  be  further  subdivided  into  events. 

These  division  points  may  be  somewhat  arbitrary  but  are  generally 
determined  as  being  at  the  point  of  completion  of  a  definite  series  or 
sequence  of  actions  for  which  a  definite  beginning  and  ending  t i me  can 
be  estab  I  i  shed .  ~ 

With  most  complex  systems  there  are  times  when  some  part  of  the 
system  is  not  functioning  properly.  However,  in  order  to  evaluate 
the  system  as  it  was  designed  to  perform  and  to  provide  a  reference 
for  comparing  degraded  performance  to  design  performance,  all  components 
of  +he  system  must  be  assumed  to  be  operatlnq  properly  in  the  creation 
of  the  evaluation  mission  and  the  operator  task  sequence.  From  this 
base  the  effect  of  various  equipment  malfunctions  upon  the  operator's 
task  and  mission  performance  may  be  evaluated. 

5.4  DERIVATION  OF  OPERATOR  TASK  LIST  AND  CREATION  OF  THE 

OPERATIONAL  SEOUENCE  DIAGRAM  (OSD) 

An  outline  of  the  mission  profile  must  be  completed  prior  to 
starting  the  operator  task  listing.  The  task  list  is  developed  within 
the  framework  of  the  mission  and  is  a  step-by-step  description  of  the 
operator's  tasks  as  the  mission  proceeds.  If  the  evaluator  has 
available  a  task  listing  for  the  design  mission  he  must  first  modify 
the  design  mission,  where  necessary,  to  a  specific  evaluation  mission 
which  is  feasible  and  representat J ve.  He  must  then  modify  the  task 
listing  according  to  this  mission.  When  no  detailed  task  listing  for 
a  design  mission  is  available  the  evaluator  will  find  it  necessary  to 
develop  both  the  evaluation  mission  and  the  task  listing  "on  the  spot" 
in  order  to  understand  the  system  in  enough  detail  to  decide  upon 
operator  performance  measurement  points.  Whether  modifying  a  desiqn 
mission  or  creating  his  own  the  operator  tasks  of  interest  to  the 
evaluator  will  be  determined  by  equipment  characteristics  and  mission 
requirements.  The  task  listinq  for  the  evaluation  mission  should 
include  only  those  which  are  mission  essential;  that  is,  those  tasks 
that  must  be  completed  successfully  in  order  to  achieve  the  prime 
mission  objective. 

i he  Operational  Sequence  Diagram  (OSD)  format,  Appendix  B, 
lists  the  operator  tasks  and  diagrams  the  interaction  of  the 
operator  with  his  equipment  and  with  other  operators.  Again,  for 
the  purpose  of  identifying  points  at  which  to  measure  operator 
performance,  the  OSD  should  contain  only  those  tasks  essential 
for  executing  the  mission.  It  should  show  when  the  operator  moves 


a  control  or  actuates  a  switch,  the  equipment  response,  the 
operator’s  observation  of  the  response  and  his  further  action. 

In  some  cases  the  operator's  response  will  be  to  verbal  instructions 
or  Information  from  other  operators.  His  action  may  be  a  manual 
action  or  a  verbal  directive. 

The  OSD  shows  qraphically  the  branching  and  feedback 
relationships  between  tasks  and  the  interactions  of  the  operator 
with  other  components.  The  illustrations  of  tne  OSD  format  given 
in  Appendix  B  and  in  MIL-H-46855  of  16  Feb  1968  are  useful  for 
learning  the  mechanics  of  the  OSD.  The  OSD  shown  in  MIL-H-46855 
uses  qeometric  symbols  to  code  actions  and  behaviors  of  the  operator 
and  his  equipment  while  the  example  in  Appendix  B  uses  letters  for 
this  code.  These  letter  codes  are  also  given  in  Appendix  B. 

The  present  writers  found  the  letter  code  to  be  somewhat 
easier  to  use  tnan  the  qeometric  symbols  in  the  qeneration  of  the 
OSD.  Tne  letter  coding  allows  also  for  the  use  of  a  computer 
qenerated  task  listinq  as  described  by  Wilson  (1968).  The  reports 
by  Wilson  (1966,  1967  and  1968)  are  highly  recommended  reading  for 
becoming  familiar  with  tne  Operational  Sequence  Diagramming  procedure. 
The  obvious  advantage  of  a  computer  generated  OSD  is  the  relative 
speed  with  which  it  can  be  changed  when  chanqas  are  made  in  the 
equipment  or  operating  procedures. 

In  creating  an  USD  it  should  be  borne  in  mind  that  its 
fundamental  purpose  is  to  depict  the  interaction  of  the  operator 
with  other  components  of  the  system.  In  this  interaction  the 
operator  makes  an  input  to  the  other  system  components  -  either 
equipment  or  other  operators.  These  components  react  in  turn  and 
information  Is  conveyed  to  the  operator  concerning  the  results  of 
his  inputs.  The  coding  and  sequential  diagramming  of  these  inter¬ 
actions  is  the  function  of  the  OSD. 

5.5  USE  OF  THE  OPERATIONAL  SEQUENCE  DIAGRAM  IN  THE  IDENTIFICATION 
OF  MEASUREMENT  POINTS 

Five  basic  steps  are  considered  necessary  for  the  measurement 
and  evaluation  of  human  operator  performance  in  the  system  during 
the  field  evaluation. 

1.  The  identification  of  manual  performance  measurement 
points,  i.e.,  the  identification  of  those  points  in 
which  the  operator  interacts  overtly  with  other 
components  of  the  system  and  at  which  quantitative 
measurement  of  his  actions  can  be  made. 

2,  Identification  of  tactical  decision  making  points 
within  the  operation  of  the  system. 
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3.  Determination  of  the  type  of  data  that  can  be 
collected  and  the  appropriate  methods  of 
record! nq  at  these  measurement  points. 

4.  Determination  of  the  performance  criteria,  i.e., 
the  level  of  performance  required  of  the  operator 
at  these  measurement  points  and  how  it  may  be 
stated. 

5.  The  evaluation  of  the  operator's  performance, 
i.e.,  the  comparison  of  his  measured  performance 
with  the  design  criteria  resulting  in  a  judgment 
as  to  the  adequacy  of  operator  performance. 

The  OSD  is  used  as  a  primary  aid  in  the  identification  of  the 
measurement  points  within  the  system  (Steps  I  &  '/).  This  may  be 
done  by  a  Human  Factors  evaluator  familiar  with  the  system  or  by 
an  operator  who  has  been  trained  for  participation  in  the  fiela 
evaluation.  If  the  Human  Factors  evaluator  has  worked  with  the 
system  during  its  development  he  would  normally  be  involved  in 
either  OSD  or  MTL  development,  be  familiar  with  tne  system  and 
hav9  already  Isolated  a  number  of  the  measurement  points.  If  the 
evaluator  has  little  experience  with  tne  system  he  may  ask  the 
trained  operators  to  identify  the  measurement  points  through 
studying  the  OSD. 

The  obiective  in  selecting  manual  performance  measurement 
points.  Is  to  identify  those  points  at  which  there  is  a  culmination 
of  activities  of  the  components  and  at  which  the  effect  of  their 
cumulative  performance  over  the  preceding  sequence  can  be  measured. 
This  may  be  conceived  as  being  a  series  of  related  activities 
funnel ing  down  to  a  point  at  which  the  operator  acts  upon  the 
Information  received.  These  are  accomplishment  points  at  which 
there  is  an  overt  and  observable  interaction  petween  the  operator 
and  tne  machine  or  between  operator  and  operator.  They  are  points 
at  which  information  flows  petween  components  and  at  which  operator 
errors  may  occur. 

The  CSD  provides  the  basic  data  for  identifying  manual 
performance  measurement  points.  It  is  recommended  to  tne 
individual  picking  tnese  points  as  the  means  for  provicinq  him 
with  a  detailed  picture  of  the  tasks  carried  out  by  the  operator. 
The  evaluator  studies  tne  OSD  and  from  his  knowledoe  and  experience 
with  the  system  selects  those  points  at  which  the  operator  overtly 
and  observably  makes  an  input  or  action  in  operating  the  system. 

The  OSD  also  provides  the  basis  for  identifying  those  points 
in  the  operation  of  the  system  at  which  decisions  important  to  tne 
success  of  the  tactical  mission  are  made.  These  decisions  may  or 
may  not  result  in  a  direct  overt  action  -  verbal  or  manual. 


Although  the  points  at  which  these  decisions  are  made  may  be 
identified  from  the  OSU  in  a  manner  similar  to  the  Identification 
of  manual  performance  measurement  points,  the  determi nat ion  of 
their  correctness,  i.e.,  the  establishing  of  a  criterion  of  adequacy 
of  the  decision,  is  generally  more  difficult  than  is  the  case  for 
performance  measurement.  This  problem  is  discussed  in  more  detail 
in  Section  5.6. 

In  carrying  out  the  studies  from  which  the  recommendat ions  in 
this  report  are  drawn  both  the  Operational  Seauence  Diaqram  and 
the  Mission  Time  Line  methods  of  analysis  were  used  on  a  trial 
basis  for  the  identification  of  measurement  points.  In  these 
trials  experienced  operators  of  the  system  were  asked  to  study 
the  analyses  and  to  indicate  performance  measurement  Doints. 

It  was  found  that  these  operators,  being  oriented  toward 
performing  tactical  operations,  tended  to  select  tactical  decision 
points  rather  than  actual  interactive  manual  Derformance  measure¬ 
ment  points.  They  tended  not  to  zero  in  on  the  more  minute 
performance  interactions  between  the  operator  and  the  equipment 
at  which  errors  could  be  observed  directly  and  measurements  of 
performance  taken.  Rather,  they  tended  to  emphasize  tactical 
decision  making  activities.  Special  and  explicit  instructions 
were  necessary  to  orient  them  toward  the  more  minute  manual 
performance  measurement  points. 

The  results  of  present  work  with  the  OSD  in  identifying 
measurement  points  underlines  the  need  to  emphasize  the  point 
tnat  experienced  operators  wilt  tend  to  focus  on  tactical  decision 
points  and  be  less  inclined  to  pick  more  minute  man-equipment 
interface  performance  points.  It  is  important  that  both  types 
of  measurement  points  be  identified.  Proper  design  of  the  points 
of  interface  will  ensure  that  errors  in  information  flow  across 
the  interface  do  not  occur.  Xowever,  it  is  also  necessary  to 
determine  whether  the  decisions  based  upon  this  information  can 
be  made  correctly.  Therefore,  both  manual  performance  measurement 
points  (points  of  man-machine  interface)  and  tactical  decision 
points  need  to  be  identified.  it  is  recommended  that  where 
possible.  Human  Factors  emineers  familiar  with  the  system 
identify  the  manual  performance  measurement  points  and  the 
identification  of  tactical  decision  points  be  made  bv  operators 
experienced  with  tne  given  or  similar  system.  If  the  experienced 
operator  must  be  called  upon  to  make  both  types  of  identifications, 
it  is  recommended  that  he  first  no  throuah  the  OSD  with  the  sole 
purpose  of  identifying  manual  performance  measurement  points. 

He  should  then  go  throunh  the  OSD  a  second  time  with  the  purpose 
of  Identifying  tactical  decision  points.  Aaain,  instructions 
emphasizing  the  distinction  between  the  two  types  of  measures  are 
especially  important  when  experienced  operators  of  the  system 
perform  the  measurement  point  i dent i f i cat  ion  exercise.  An 
illustration  of  tne  format,  instructions  and  results  for  such 
an  exercise  is  given  in  Appendix  C. 
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In  making  his  Identification  of  performance  points  the 
evaluator  or  operator  should  avoid  making  judgments  as  to  whether 
the  points  he  selects  are  critical  to  system  operation.  He  should 
aporoach  the  task  of  Identi fylng  points  with  the  orientation  that 
every  action  in  a  system  is  necessary  to  the  success  of  the 
system.  When  all  measurement  points  have  been  identified  they 
may  be  rated  with  respect  to  relative  criticality  if  it  Is  found 
that  it  is  not  practical  to  record  or  observe  at  ail  points 
identified.  However,  where  possible  performance  records  should 
be  taken  at  all  measurement  points  which  have  been  identified. 

5.6  PERFORMANCE  RECORDING  AND  EVALUATION 

After  identifying  the  points  in  the  operation  of  the  system  at 
wnich  performance  measurements  may  be  taken,  the  Human  Factors 
evaluator  must  decide  what  measurements  to  take.  In  what  testing 
situation  to  take  them  and  how  to  evaluate  the  measures  once  they 
are  collected. 

In  determining  what  measurements  to  take  the  HFE  must  decide 
Doth  the  kind  of  behavior  to  be  recorded  and  the  particular  method 
or  technique  for  recording  it.  He  must  also  decide  upon  a  scale 
by  which  he  will  quantify  the  behavior  and  he  must  determine  the 
criterion  or  level  of  performance  required.  In  this  section  four 
task  types  are  descriDed  and  scales  of  measurement,  criteria, 
recording  techniques,  the  evaluative  judgment  appropriate  to  eacn, 
and  test  conditions  are  discussed. 

5.6.1  Task  Types 

The  evaluator  may  distinguish  three  different  types  of  manual 
task  performance  which  require  somewhat  different  measurement 
scales,  criteria  and  methods  of  recording.  These  are  (I)  continuous 
control  activities  in  which  the  operator  tracks  a  moving  taraet  or 
nulls  an  error  through  continuous  control  movement,  (2)  discrete, 
control  activities  such  as  pressing  a  button  or  throwing  a  switch 
and  (3)  positioning  a  pointer  or  marker  with  reference  to  some 
index  such  as  setting  a  dial  or  positioning  a  symbol  on  a  scope 
face. 

The  fourth  task  type  which  the  HFE  must  consioer  is  that  of 
tactical  decision  making  already  mentioned  in  Section  5.5.  These 
decisions  may  be  of  the  obvious  simple  yes-no-variety.  For  those 
decisions  which  appear  to  be  highly  complex  or  appear  to  require 
choosing  among  a  number  of  alternatives  the  evaluator  should 
analyze  the  sequence  of  events  to  determine  whether  the  seemingly 
complex  set  of  options  cannot  be  reduced  to  a  series  of  simple 
"yes-no"  decisions.  Often  they  can. 


5.6.2  Measurement  Scales 


In  selectinq  measurement  scales,  reference  to  the  detailed 
description  of  the  tasks  of  the  Operational  Sequence  Diaqram 
will  aid  tne  evaluator  in  making  decisions  about  what  type  of 
scale  is  appropriate  for  each  task.  Table  I,  Column  2  summarizes 
briefly  the  scales  appropriate  for  the  different  task  types. 

5.6.3  The  Cr iter i a 

With  the  identi f ication  of  measurement  points  within  the 
system,  it  is  necessary  to  determine  whether  criteria  are  available 
for  these  points.  These  criteria  must  set  forth  the  tolerances  or 
limits  of  required  performance  in  the  scale  of  measurement  appro¬ 
priate  to  the  performance. 

The  determination  of  the  criterion  level  of  performance  at 
the  selected  manual  performance  measurement  points  will  usually 
require  choosing  among  several  different  approaches.  Theoretically 
(and  ideally)  the  system  designers  specify  the  accuracy  with  which 
the  human  operator  is  required  to  perform.  Usually,  this  ideal 
is  only  partially  attained. 

At  those  measurement  points  at  which  categorical  binary 
decisions  or  discrete  manual  actions  are  required  the  criterion 
conditions  can  usually  be  stated  directly,  i.e.,  the  operator  is 
expected  to  exercise  the  correct  option  each  time.  At  those 
points  at  which  a  distribution  of  errors  is  possible  neither 
the  allowable  limit  of  error  nor  the  expected  error  distribution 
may  be  stated  explicitly. 

The  Human  Factors  evaluator  may  obtain  useful  information  as 
to  allowable  error  limits  by  questioning  the  system  design  engineers 
and/or  through  his  own  further  detailed  analysis  of  the  system. 

For  each  identifiable  manual  performance  measurement  point  he  must 
ask  "what  are  the  limits  of  accuracy  within  which  this  task  must 
be  performed  in  order  for  the  mission  to  succeed?'  For  some 
measurement  points  he  will  not  be  successful  in  defining  the  limits. 
For  these  the  deslqn  engineer  may  have  made  a  judgment,  based  upon 
his  experience  in  designing  similar  systems,  that  the  operator 
would  be  able  to  perform  adequately  and  that  the  effectiveness  of 
the  system  would  increase  with  operator  proficiency. 

When  no  explicit  criteria  are  available  for  a  qiven  measurement 
point  it  is  necessary  to  record  the  performance  of  the  operator  at 
that  point  in  "raw  data"  form  without  reference  to  criteria. 

These  raw  data  provide  the  basis  for  estimating  the  effect  of 
operator  performance  at  that  point  uDon  total  system  performance. 

An  estimate  of  the  reliability  of  the  operator  In  performing  at 
that  point  is  obtained  which  can  be  used  in  computing  total  system 
reliability  estimates. 
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TABLE  I. 

SUMMARY  OF  TASK  TYPE,  SCALE  OF  MEASUREMENT,  CRITERIA  AND  EVALUATIVE  YARDSTICK 
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In  collecting  "raw  data"  as  well  as  collecting  data  relative  to 
a  criterion  it  is  necessary  that  the  performance  be  measured  repeatedly 
under  a  given  set  of  conditions  so  that  a  reasonably  reliable  data 
pase  may  be  obtained.  The  goal  is  to  obtain  as  accurate  an  estimate 
of  the  real  distribution  of  errors  as  possible  through  sampling  the 
performance  of  a  number  of  operators.  Of  course,  the  larger  the 
number  of  operators  sampled  on  a  given  task  under  a  given  set  of 
conditions  the  better  the  estimate  (or  prediction)  cf  what  the 
performance  will  be  when  the  system  is  operated  in  tne  future.  In 
carrying  out  tests  for  purposes  of  predicting  future  performance 
it  is  important  also  that  the  conditions  of  test  and  the  operators 
tested  should  be  as  representative  as  possible  of  those  to  which 
prediction  is  to  pe  made. 

In  the  field  test  situation,  difficulty  may  be  experienced  in 
making  repeated  tests  under  the  same  conditions.  This  is  particularly 
true  of  testing  in  the  fully  operational  system  since  it  is  diffi¬ 
cult  to  maintain  standard  testing  conditions  from  test  run  to  test 
run.  Tne  use  of  other  test  situations  such  as  a  simulator  of  the 
actual  operational  system  has  certain  advantanes  for  repeating 
standard  test  conditions  and  accumulating  data  over  a  greater  number 
of  test  runs.  The  use  of  simulators  as  well  as  other  test  situations 
are  discussed  in  Section  b.7.  Whenever  it  is  possible  to  measure 
performance  reliably  in  the  actual  system  and  to  take  measures  durirn 
at  least  b  identical  test  runs,  a  reasonable  data  base  for  predicting 
future  performance  will  have  been  obtained.  For  a  review  of  the 
techniques  of  samplina  and  of  estimating  population  parameters  see 
Johnson  (1949,  pp.  104-117),  and  Cochran  (1964,  pn.  18-26). 

The  criterion  or  acceptaDle  level  of  performance  for  tactical 
decisions  is  often  difficult  to  define.  The  correctness  of  a 
decision  at  a  given  point  in  the  system  operation  may  not  be 
ascertainable  until  a  later  time  after  subsequent  events  have 
unfolded,  tvents  and  decisions  subsequent  to  the  decision  in 
question  may  make  tne  assessment  of  its  correctness  virtually 
impossible.  Some  success  may  be  attained  in  breaking  down  the 
decision  process  into  a  component  series  of  ves-no  decisions  each 
of  which  can  oe  evaluated  as  to  its  correctness.  Where  a  playback 
record  of  the  events  of  tne  test  run  is  available,  as  for  example 
in  tne  P-36  A pv»  system,  a  panel  of  experts  may  view  tne  playback 
and  judge  the  adequacy  of  trie  decision.  More  is  said  concern  inq 
this  problem  in  Section  P.6.7  in  which  tne  measurement  and  evaluation 
of  tactical  decisions  is  covered. 


5.6.4  Measurement  and  Evaluation  of  Continuous  Control  Tasks 
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Continuous  control  tasks  may  be  of  two  types.  One  Is  termed 
compensatory  control  In  which  the  operator  seeks  to  hold  an 
Indicator  In  alignment  with  some  fixed  index  and,  through  moving 
the  control,  compensate  for  disturbances  which  tend  to  move  the 
indicator  away  from  the  index.  An  example  of  a  compensatory  control 
task  Is  the  holding  of  a  given  aircraft  bank  angle  using  the 
attitude  indicator.  The  second  type  of  continuous  control  task  is 
the  pursuit  task  in  which  the  operator  seeks  to  hold  an  Indicator 
in  alignment  with  an  index  which  is  moving.  An  example  of  this  type 
of  task  Is  the  positioning  of  a  symbol  over  a  movino  target  on  a 
radar  scope  through  movement  of  a  control. 

In  continuous  control  tasks  the  measurement  scale  problem  is 
that  of  choosing  an  appropriate  metric  (inches,  degrees,  etc.)  by 
which  the  amount  of  deviation  of  the  Indicator  from  the  index  may 
be  guantifled.  These  deviations  are  measured  and  recorded  from 
moment  to  moment  and  provide  a  guantitative  expression  of  task 
performance.  Comparison  of  this  performance  with  the  criterion, 
or  reguired  level  of  performance,  is  the  essence  of  the  evaluation 
process. 

Tne  criterion  or  reguired  level  of  performance  may  be  given 
in  terms  of  (I)  a  standard  to  be  held  such  as  a  given  altitude, 
or  (2)  limits  about  a  standard  within  which  the  system  is  to  be 
controlled,  e.g.,  ±  50  feet  of  altitude.  As  indicated  in  Table  I 
the  type  of  measurement  taken  (measurement  scale)  will  be  different 
for  the  two  types  of  criteria.  When  the  criterion  is  given  in 
terms  of  holding  a  standard  the  appropriate  measurement  scale  wi II 
express,  through  some  numerical  value,  the  amount  of  deviation  or 
error  about  the  standard.  When  criterion  limits  have  been  set  up 
about  some  standard,  the  amount  or  percent  of  time  outside  those 
limits  can  be  recorded  as  a  numerical  expression  of  performance. 

For  tnese  measures  either  a  distribution  of  errors  or  time  out  of 
limits  for  all  operators  over  all  test  runs  can  be  generated. 

When  no  time  out  of  limits  or  no  errors  larger  than  the  criterion 
limit  occur,  the  evaluative  decision  is  straightforward.  The 
operator  is  performing  to  the  criterion.  If  the  error  distribution 
contains  errors  which  are  greater  than  tne  criterion  limit  the 
freguency  or  percentage  of  these  errors  form  the  basis  for 
'predicting  the  magnitude  and  frequency  expected  during  future 
operation  of  the  system  and  for  calculating  their  effect  on  total 
system  performance.  The  disposition  of  tne  case  in  which  performance 
outside  of  limits  occurs  may  take  a  number  of  forms.  The  problem 
may  be  corrected  by  redesign  of  equipment,  further  traininq  or  re¬ 
training  of  operators,  or  chanqes  in  procedures  in  which  the  role 
of  the  operator  is  reordered. 
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For  measuring  deviations  from  the  standard  or  for  determining  /> 

time  out  of  limits  for  continuous  motor  tasks  it  is  desirable  to 
use  some  method  for  recording  error  continuously.  In  some  systems 
traces  or  records  of  parameters  of  performance  of  the  system  may 
be  obtained  since  they  are  generated  and  used  as  a  part  of  the 
operation  of  the  system.  For  example,  in  the  P-3C  ASW  system 
various  aircraft  performance  parameters  are  sensed  and  used  in  the 
solution  to  the  ASW  problem.  Some  form  of  magnetic  tape  recording 
is  optimum  for  recording  such  parameters  wnen  summarization  and 
analysis  of  the  data  can  be  carried  out  using  ground  based  computers. 

When  limits  criteria  are  given,  a  hard  copy  pen  recording  of 
performance  may  pe  adeauate  since  a  template  or  limit  lines  may  be 
used  for  ootaininq  measurements  of  time  out  of  Mr  its  with  a  fair 
degree  of  ease.  A  less  desirable  method  of  recording  is  a  film 
record  of  the  operator's  instruments  or  displays  from  which  the 
evaluator  may  sample  and  measure  tne  error  over  the  period  of  tne 
performance.  Finally,  the  evaluator  may  act  as  an  observer  and 
merely  sample  tne  performance  on  a  periodic  basis  and  record  it 
manually.  This  latter  method  is  quite  satisfactory  when  the 
deviations  of  the  system  from  the  standard  are  of  low  frequency, 
i.e.,  error  accumulates  slowly,  so  that  the  observer  has  adequate 
time  to  sample  and  record  the  performance  at  periodic  intervals. 

The  photographic  and  observation  techniques  can  be  compined  when 
motion  film  of  the  performance  is  taken  and  later  reviewed  by  one 
or  more  evaluators.  During  this  review  observations  may  be  made 
at  pre-set  periodic  intervals.  This  combination  is  often  the  most 
expeditious  and  least  expensive  method  for  obtaining  continuous 
manual  performance  data.  It  provides  hard  records  of  performance 
with  the  least  expense  and  often  with  the  least  interference  with 
the  operation  of  the  system. 

5.6.5  Measurement  and  Evaluation  of  Discrete  Manual  Tasks 

Discrete  manual  tasks  are  those  in  which  the  operator  makes  a 
discrete  motion  either  to  push  a  button  or  to  position  a  multiple 
position  switch.  The  recording  problem  here  is  one  of  recording 
an  event.  The  criteria  may  be  of  two  types.  In  the  first  the 
button  is  either  pressed  or  the  switch  is  positioned  correctly. 

In  the  second  tne  button  or  switch  is  activated  in  the  correct 
sequential  position  relative  to  other  discrete  motor  acts. 

The  recording  problem  with  respect  to  discrete  tasks  is  some¬ 
what  simpler  than  the  problem  of  continuous  motor  performance 
recording.  The  discrete  action  may  be  recorded  as  an  event  either 
on  magnetic  tape  or  on  a  hard  copy  osc i I  I oaraph i c  record.  The 
accuracy  or  appropriateness  of  the  action  can  be  determined  throuah 
either  computer  processing  of  the  tape  record  or  visual  inspection 
of  the  hard  copy.  With  this  type  of  task  a  film  or  video  tape 
record  is  a  useful  recording  technique  since  the  evaluator  can 
determine  throuqh  oDservinq  the  recording  whether  the  actions  were 
correct  and  in  the  proper  sequence. 
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A  particular  advantage  of  the  film  or  video  tape  record  is  the 
coverage  of  a  number  of  controls  through  one  recording  medium.  The 
opportunity  to  observe  and  evaluate  the  performance  away  from  the 
actual  working  system  and  to  re-play  the  action  is  also  advantageous. 
The  solution  to  the  problems  of  positioning  cameras  and  the  technical 
problems  of  obtaining  readable  film  are  difficult  in  some  systems. 
However,  where  possible  of  being  used,  the  photographic  or  video 
techniques  have  much  to  recommend  them  since  they  capture  a  great 
deal  of  data  for  later  reduction,  analysis  and  study. 

With  the  discrete  manual  tasks  and  the  dial  setting  tasks 
discussed  earlier,  direct  observation  and  manual  recording  by  the 
evaluator  often  may  be  tne  most  expeditious  and  satisfactory  means 
of  recording  performance.  When  the  evaluator  is  quite  familiar  with 
the  task  demands  and  is  practiced  in  observing  them  he  can  cover  a 
great  many  discrete  and  dial  setting  task  performances  quite  reliably. 

5.6.6  Measurement  and  Evaluation  of  Dial  Setting  Tasks 

As  with  the  discrete  manual  tasks  the  dial  settinq  tasks  are 
discrete  events  which  can  be  recorded  as  events  either  throuqh  use 
of  recording  equipment  or  through  observation  by  tne  evaluator.  Tne 
criterion  may  be  either  a  required  dial  setting  or  the  carrying  out 
of  the  event  in  the  correct  seauence  witn  respect  to  other  discrete 
events  within  a  series  of  events.  Recording  may  be  accomplished 
either  throuqh  recording  on  magnetic  or  hard  copy  taoe  or  through 
film  or  video  recordings.  In  general  they  may  be  treated  as  discrete 
motor  tasks  except  that  there  may  be  specific  requirements  with 
respect  to  the  accuracy  with  which  the  dial  must  be  set.  In  this 
respect  they  are  like  the  continuous  tracking  tasks  in  that  the 
frequency  of  settings  outside  of  limits  must  be  ascertained. 

5.6.7  Measurement  and  Evaluation  of  Tactical  Decisions 


The  recording  of  tactical  decisions  is  quite  similar  to  recording 
discrete  manual  tasks  when  the  decisions  are  observable  as  overt 
verbal  or  motor  acts.  The  complication  in  evaluating  such  overt 
decisions  comes  from  the  difficulty  in  establishing  the  criteria  as 
to  their  correctness.  However,  the  tactical  decision  mav  be  "not  to 
act."  In  this  case  the  fact  of  the  decision  and  its  degree  of 
appropriateness  may  be  evident  only  after  the  occurrence  of  a  number 
of  subsequent  events. 

To  further  complicate  the  evaluation  of  tactical  decisions, 
they  are  made  in  a  dynamic  evolving  situation  in  which  tne  antecedent 
conditions  to  the  decision  are  almost  never  the  same  from  one  test 
run  to  another.  This  is  particularly  the  case  when  evaluating  in 
the  actual  operating  system.  Antecedent  conditions  during  evaluations 
can  be  controlled  much  more  satisfactorily  in  tne  dynamic  simulator 
of  the  system. 
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In  making  judgments  as  to  the  correctness  of  tactical  decisions 
it  should  be  emphasized  that  the  correctness  of  the  decision  is  a 
function  of  both  the  accuracy  of  the  data  received  by  the  decision 
maker  and  the  appropriateness  of  his  interpretation  and  analysis  of 
the  data.  For  example,  in  the  ASW  system  an  incorrect  tactical 
decision  may  come  about  as  a  result  of  improper  assessment  of  the 
situation  on  tne  part  of  the  Tactical  Coordinator  or  as  a  result  of 
inaccurate  information  supplied  to  him  by  other  components  of  the 
system. 

Further,  the  assessment  of  the  correctness  of  a  qiven  decision 
"on-tne-spot"  at  the  time  it  is  made  is  often  not  appropriate  since 
the  validity  of  the  decision  must  be  established  in  the  light  of 
the  events  following  it.  Knowledge  of  the  Information  upon  which 
the  decision  was  made,  the  decision  itself  and  tne  effect  of  the 
decision  upon  mission  success  are  all  necessary  to  judging  its 
adequacy. 

Within  some  systems  the  events  surrounding  the  makina  of  the 
tactical  decision  may  be  recorded  in  enouqh  aetai I  to  make  re¬ 
construction  or  "play-back"  of  tne  events  possible  outside  the 
operation  of  tne  system,  under  these  conditions  expert  observers 
may  make  a  judgment  as  to  tne  correctness  of  the  decision.  In  the 
P-3C  ASW  system,  for  example,  a  number  of  records  are  taken  which 
can  be  played  back  for  use  in  determining  the  adeauacy  of  tactical 
decisions.  In  other  systems  the  records  taken  to  evaluate  the 
performance  of  the  system  equipment  may  be  "played  back"  and 
analyzed  to  determine  the  correctness  of  operator  decisions  made 
during  the  evaluation  mission. 

Any  knowledge  of  operator  action.  Instrument  information  and 
verbal  interchange  can  be  used  to  advantage  by  exDert  observers  to 
arrive  at  judgments  as  to  the  correctness  of  decisions.  Film  and 
video  recordinas,  sound  recordings  and  magnetic  or  other  recordings 
of  events  and  performance  must  be  used  to  the  extent  available  in 
evaluatinq  the  tactical  decision.  Examination  of  the  Ooerational 
Seauence  Ulaqram  to  identify  the  points  of  information  exchanqe  and 
points  of  decision  is  recommended  as  beim  essential  to  the 
determi nation  of  points  of  measurement,  types  of  recordina  and 
scales  of  measurement. 

When  tactical  decisions  are  judged  to  be  in  error  it  is  necessary 
to  examine  the  system  both  with  respect  to  the  adequacy  of  the  equipment 
design  and  the  complexity  and  sequencing  of  the  data  whicn  the  operator 
is  required  to  interpret  and  analyze.  As  indicated  above,  two 
sources  of  variability  contributing  to  decision  error  may  be 
identified  -  information  error  and  assessment  error.  A  re¬ 
examination  of  the  task  descriptions  in  the  Operational  Sequence 
Diagram  and  the  manner  in  which  they  are  sequenced  may  reveal  desiqn 
inadequacies  which  are  contributing  to  the  tactical  decision  error. 


Tha  evaluator  may  find  it  necessary  to  set  up  tests  In  other 
environs  such  as  the  laboratory  or  simulator  in  order  to  Isolate 
the  contribution  of  information  error  to  the  total  tactical 
decision  making  error. 
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b.7  TESTING  CONDITIONS 


I 


What  are  the  test  conditions  under  which  operator  performance 
measurements  may  be  taken?  During  field  test  and  evaluation  the 
goal  Is  to  obtain  information  about  the  effectiveness  of  the  system 
under  conditions  representative  of  its  functioning  In  operational 
use.  From  tnese  data  a  forecast  ray  be  made  about  how  the  system 
will  function  in  the  operational  environment.  Therefore,  the 
Human  Factors  evaluator  attempts  to  obtain  data  on  operator 
performance  under  conditions  which  will  be  most  predictive  of  Its 
performance  durinq  operations.  The  validity  of  his  data  will 
depend  both  upon  how  well  his  test  situation  represents  the 
operational  conditions  and  upon  the  reliability  and  discriminating 
power  of  his  measurements.  The  degree  to  which  the  test  conditions 
can  be  representative  of  operational  conditions  and  the  reliability 
of  the  measurements  taken  wiil  vary  greatly  from  system  to  system. 

The  complexity  of  the  system  and  the  facilities  and  personnel  made 
available  for  the  test  will  oe  important  influences  on  the  adequaev 
of  the  tests. 

Often  it  will  be  necessary  to  compromise  between  test  conditions 
wnich  provide  reliable  measures  under  repeatable  conditions  and 
those  which  provide  the  most  representative  operational  conditions. 

In  qeneral,  the  greater  the  number  of  operational  parameters 
incorporated  into  the  test  situation  the  more  difficult  it  is  to 
maintain  standard  test  conditions  from  test  run  to  test  run. 

In  like  fashion,  the  more  control  exercised  over  the  test  situation 
in  order  to  attain  repeataDle  conditions  the  more  unlike  the 
operational  conditions  it  may  become.  This  is  not  a  necessary 
state  of  affairs  but  generally,  the  more  variables  operating  in  the 
situation  to  whicn  one  wishes  to  predict  the  more  difficult  it  is 
to  control  them  so  that  standard  test  conditions  may  be  attained 
across  a  series  of  test  trials. 

In  the  operation  of  many  military  systems  many  important 
variables  are  simply  not  controllable  and/or  man i du I atab le  within 
the  test  situation.  Variables  such  as  wind  speed,  turbulence  and 
visibility  may  affect  some  systems  so  that  controlled  test  conditions 
are  impossible.  In  such  cases  the  evaluator  must  obtain  an  assessment 
of  the  level  of  each  of  these  variables  durinq  the  test  run  so  that 
tneir  effect  upon  oerformance  may  be  estimated  -  through  calculation 
of  correlation  coefficients  where  sufficient  data  are  obtained  or 
subjectively  where  it  is  not. 

The  Human  Factors  evaluator  should  bear  in  mind  that  it  will  be 
of  I i mi  ted  va I ue  to  concuct  tests  in  a  highly  realisTic  and 
representative  test  situation  if  it  is  not  possible  to  obtain 
repeated  measures  of  task  performance  under  the  same  test  conditions. 
On  the  other  nand  reliable  measures  taken  in  a  non-representative 
test  situation  are  eaually  limited. 
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Keeping  in  mind  that  he  wishes  to  maximize  his  predictions  of 
actual  operational  performance  from  his  test  data  the  evaluator 
should  consider  at  least  four  different  test  situations  as  sources 
of  evaluation  data.  These  are  (I)  the  actual  operating  system, 

(2)  dynamic  simulators  of  the  system,  (3)  test  ranges,  and  (4) 
special  I aDoratory- I i ke  assemblies  of  subsystems  and  parts,  or 
simulators  of  pjris,  of  the  system.  These  test  situations  are 
discussed  in  tne  following  sections. 

3.7.1  The  Operating  System  as  a  Test  Situation 

The  test  condition  most  representative  of  the  operational 
use  of  the  system  during  wnich  evaluation  data  may  be  collected 
is  that  in  which  the  full  operating  system  is  carrying  out  the 
standard  evaluaTion  mission.  In  the  actual  conduct  of  human  factors 
evaluation  during  this  mission  the  performance  measurement  may 
require  that  the  testing  be  fitted  into  the  test! no  of  the  system 
equipment,  in  many  instances  human  factors  test  and  evaluation 
can  be  worked  out  satisfactorily  unaer  these  conditions.  However, 
the  scheduling  of  operational  runs  for  the  prime  purpose  of  obtaining 
human  factors  data  will  be  necessary  for  fully  adequate  conduct  of 
such  testing. 

While  operation  of  the  actual  system  is  the  most  representative 
condition  during  which  to  measure  performance  the  problems  to  be 
encountered  should  be  appreciated.  The  major  deterrent  to  good  test 
procedure  is  the  difficulty  in  obtaining  measures  under  repeated 
test  conditions  in  which  the  important  variables  in  the  testing 
environment  are  kept  the  same  from  test  trial  to  test  trial.  When 
the  test  conditions  vary  from  trial  to  trial  a  reliable  picture 
of  operator  performance  cannot  be  obtained.  Without  reliable 
measures  of  operator  performance  under  the  test  and  evaluation 
conditions  valid  predictions  of  how  he  will  perform  in  the  operational 
situation  are  not  possible.  Further,  unless  such  reliable  measures 
can  be  obtained  in  actual  full  system  operation  the  validity  of  the 
evaluative  decisions  made  earlier  in  tne  development  of  the  system 
cannot  be  determined.  With  attention  to  the  requirements  for 
reliable  testinq  during  tne  planning  and  staffing  for  field  test 
and  evaluation  the  repeatability  of  test  conditions  can  be 
adequately  assured  for  most  systems.  The  Hunan  Factors  evaluator 
must  remember  simply  that  he  must  adhere  as  closely  to  good 
experimental  procedure  as  is  possible  under  field  conditions. 

This  entails  the  establishment  of  a  test  plan  which  incorporates 
measuring  performance  under  standard  test  conditions  over  er.ouqh 
test  runs  to  be  assured  that  an  accurate  estimafe  of  the  true 
performance  of  tne  operator  under  those  conditions  is  obtained. 

Since  the  equipment  evaluator  must  be  concerned  with  the  same 
problems  the  planning  of  the  test  runs  to  meet  the  needs  of  both 
the  eauipment  and  the  operator  evaluator  will  be  possible  for 
much  of  the  test  inn. 


Where  control  over  test  conditions  is  not  possible  in  the 
field,  the  systematic  observation  and  the  measurement  of  operator 
performance  will  serve  to  identify  those  measurement  points  at 
which  extreme  variability  in  performance  is  occurring.  These  tasks 
may  then  be  examined  to  determine  whether  (I)  an  obvious  equipment 
design  or  procedural  problem  is  present  or  (2)  the  measurement 
of  performance  at  this  point  might  be  profitably  undertaken  in 
another  test  situation  such  as  the  simulator,  test  range  or  laboratory. 

5.7.2  The  Use  of  the  Simulator  for  Test  and  Evaluation 

In  usinq  the  simulator  as  an  evaluation  tool  the  evaluator 
must  be  particularly  aware  of  its  limitations  with  respect  to  its 
representing  the  total  system  and  its  conditions  of  operation. 

On  the  one  hand  the  simulator  may  not  incorporate  many  of  the 
tasks  and  conditions  of  the  actual  operating  system.  On  the 
other,  it  allows  for  the  repetition  of  standard  test  procedures 
and  the  introduction  of  variables  and  conditions  into  the  test 
situation  which  miqht  not  be  feasible  or  safe  in  the  actual 
system.  Therefore,  the  informed  use  of  the  dynamic  simulator 
of  the  system  for  obtaining  operator  performance  data  during 
repeated  runs  of  the  mission,  or  seqments  of  the  mission,  must 
be  considered  a  valuable  evaluation  method. 

Two  test  plans  were  used  by  the  present  authors  in  field  tests 
to  Investigate  the  use  of  a  system  trainer  as  an  operator  performance 
evaluation  tool.  The  particular  trainer  used  in  this  instance  was  the 
P-3C  Weapon  System  Trainer  (NAVTRADEVCEN  Device  2F87).  In  the  conduct 
of  these  tests  two  different  lengths  of  test  run  were  used.  The 
first  test  employed  short  segments  of  the  mission  so  that  many 
repetitions  of  the  run  could  be  carried  out  during  a  test  period. 
Objections  to  tne  short  mission  segment  were  voiced  by  the 
experienced  operators  used  as  subjects  for  the  test.  They  expressed 
a  lack  of  interest  in  repetition  of  rne  short  mission  segment  and 
were  not  able  to  fully  appreciate  the  evaluators  need  to  obtain  an 
adequa+e  sample  of  performance  scores.  The  second  test  of  the 
simulator  as  an  evaluation  tool  employed  a  much  more  inclusive  set 
of  conditions  representing  a  major  portion  of  an  ASW  mission.  These 
test  conditions  were  much  more  acceptable  to  the  subject  operators 
than  was  the  short  test. 

The  lenath  of  the  test  run  and  the  number  of  variables  and 
conditions  to  be  included  must  be  decided  by  the  evaluator  on  the 
basis  of  his  knowledge  of  good  test  conditions,  his  experience 
in  dealing  with  the  constraints  of  the  field  test  situation 
including  the  limitations  of  the  trainer,  and  his  ability  to 
interest  and  motivate  the  experienced  operators  performing  the 
task.  He  should  make  it  a  point  to  enlist  the  operators’ a i d  by 
informing  them  of  the  purpose  of  the  tests  and  assuring  them  that 
they  are  not  being  evaluated  as  individuals  -  rather  they  are 
assisting  in  the  evaluation  of  the  system. 
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0.1.5  Test  names  ds 


Certain  tasks  may  ue  performed  on  test  ranges  ur  -;nr  <.j,\  : 
which  can  be  repeated  ano  performance  data  accumulated.  in  sue; 
cases  the  actual  ope~atinq  system  may  oe  used  but  only  a  certain 
segment  of  the  mission  be  carried  out.  Data  obtained  under  such 
test  conditions  can  be  collected  under  quite  representative  test 
conditions  using  measurement  equipment  which  provides  highly 
reliable  data. 

As  an  example  of  the  use  of  ranqes,  the  visual  "mark-on-top" 
task  of  the  P-3C  pilot  is  a  task  in  which  accuracy  of  performance 
might  well  have  been  tested  on  an  instrumented  range  as  a  single 
task  with  empirical  performance  data  collected.  The  error 
distributions  obtained  under  such  conditions  may  be  used  to 
determine  the  contribution  of  measured  pilot  error  to  overall 
system  error.  The  data  obtained  during  repeated  runs  on  the 
test  range  using  representative  operators  can  often  give  a  more 
reliable  estimate  of  operator  performance  than  would  be  possible 
if  the  total  mission  test  were  used. 

Test  range  data  as  well  as  the  laboratory  test  data  discussed 
below  should  play  a  large  part  in  the  evaluative  decisions  made 
during  the  development  phase  of  the  system.  However,  the  field 
evaluator  may  f I nd  both  the  test  range  and  the  laboratory  very 
useful.  Usually  he  will  have  the  advantage  of  greater  knowledge 
of  the  tactical  employment  of  the  system  as  delivered  and  access 
to  more  representative  operators  with  which  to  test  the  system. 

The  data  he  collects  will,  therefore,  reflect  a  more  accurate 
picture  of  operational  performance  than  will  the  earlier  development 
data. 

5,7.4  Laboratory  Tests 

As  with  the  simulator  and  range  tests,  for  some  tasks  it  may 
be  more  appropriate  to  test  the  performance  of  the  operator  under 
quite  closely  controlled  test  conditions.  The  operator's  ability 
to  detect  and  respond  to  certain  signals  or  inputs  may  be  best 
tested  in  a  controlled  situation  under  the  assumption  that  if  he 
cannot  perform  satisfactorily  under  such  conditions  the  probabilities 
are  against  his  doing  so  In  the  complex  operational  system.  The 
identification  of  such  problem  areas  will  come  from  operation  of 
the  actual  system.  In  the  collection  of  definitive  data  regarding 
them  the  laboratory  test  situation  may  often  be  appropriate. 

5.8  WORKLOAD  ANALYSIS 

An  important  evaluative  question  with  respect  to  man's 
performance  in  the  system  is  whether  he  has  the  time  necessary 
to  complete  the  actions  and  decisions  required  of  him  throughout 
the  course  of  the  mission.  What  is  his  workload  and  is  It 
excessive  at  any  time? 
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Durina  the  development  of  the  system  the  operator's  workload 
will  have  been  considered  and  assessed  either  formally  and 
objectively  or  by  subjective  estimate.  If  formally  assessed, 
the  method  used  will  most  probably  have  employed  the  Operational 
Sequence  Jiaqram  or  Mission  Time  Line  to  detail  the  tasks;  have 
estimated  the  time  required  to  execute  each  of  the  tasks; 
determined  time  available  for  completion  of  each  of  the  several 
seqments  of  the  mission;  and  compared  +ime  required  to  time 
available  as  a  quantitative  estimate  of  workload.  At  the  field 
evaluation  level  tne  evaluator  must,  throuqh  observation  of 
operational  performance,  determine  whether  the  tasks  required 
durinq  any  mission  segment  are  performed  in  their  proper  sequence 
within  the  time  available.  Fallinq  behind  In  the  execution  of 
tasks  or  omitting  tasks  from  the  sequence  Indicate  an  excessive 
workload  on  the  operator.  When  such  conditions  are  observed 
one  of  the  first  questions  the  evaluator  should  consider  is 
wnetner  the  experience  of  the  operator  being  observed  is 
equivalent  to  that  of  the  operational  system  operator.  It  may 
be  possible  to  conduct  repeated  runs  of  the  particular  mission 
segment  in  the  simulator  to  determine  the  level  of  learning, 
learning  curve  and  asymptotic  performance  of  the  operators.  He 
can  also  gain  an  estimate  of  the  effect  of  fatique  upon  the 
performance  and  timely  execution  of  the  task. 

5.9  EVALUATION  OF  THE  DESIGN  FOR  OPERATOR  FEEDBACK 


For  most  control  actions  carried  out  durinq  the  operation  of 
a  system  the  results  of  the  action  are  readily  apparent  to  the  operator. 
That  is  to  say  that  there  is  immediate  feedback  or  knowledge  of 
results  of  his  inputs  to  the  system.  However,  the  system  may  be 
so  designed  that,  followinq  an  operator  action,  a  lengthy  sequence 
of  system  events  unfolds  before  it  is  apparent  to  him  whether  or 
not  the  action  was  correct.  His  action  may  be  one  which  commits 
the  system  to  a  qiven  tactic  and,  if  incorrect,  will  lead  him  to 
proceed  on  the  assumption  that  the  system  is  operating  in  a  certain 
way  when,  in  fact,  it  is  not.  By  the  time  the  result  of  the 
action  error  is  apparent  tne  mission  may  be  irretrievably  compromised. 

The  Human  Factors  evaluator  should  examine  the  Operational 
Sequence  Diagram  (or  the  Mission  Time  Line  if  the  task  analysis  is 
in  that  form)  and  for  each  operator  Input  to  the  system  ask  the 
following  questions:  (I)  Does  the  operator  receive  feedback  as  to 
the  accuracy  of  his  actions?  (2)  If  he  receives  feedback,  does  he 
receive  it  rapidly  enough  to  correct  an  error  before  the  mission 
is  seriously  compromised?  If  the  operator  receives  no  feedback, 
if  it  is  delayed  or  if  it  is  at  all  ambiguous  a  human  factors 
design  problem  exists.  Early  examination  of  the  task  analysis  to 
identify  such  problems  may  well  save  discovery  of  a  design  deficiency 
before  costly  performance  measurement  has  been  undertaken. 
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5.10  THE  PERSISTENCE  OF  EARLIER  PROCEDURES  AND  HABITS 


If  the  evaluator  observes  the  operator  using  procedures  and 
methods  carried  over  from  an  earlier  system  but  not  designed  to 
be  used  wltn  the  new  he  should  inquire  further  Into  the  matter. 
Almost  every  system  is  an  evolutionary  step  from  some  predecessor. 
Personnel  chosen  to  operate  the  new  system  during  Its  evaluation 
will  be  those  who  are  highly  skilled  with  the  ancestral  system. 

If  they  persist  In  using  habits  and  methods  appropriate  to  the 
earlier  system  in  performing  with  the  new,  a  design  problem  may 
well  be  indicated.  The  evaluator  should  understand  that  the 
numan  being  has  a  proclivity  for  carrying  old  habits  over  into 
new  situations.  However,  when  old  methods  are  persistently  used 
In  conflict  with  those  called  for  by  the  design,  the  probability 
of  a  design  fault  should  be  Investigated. 
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6.0  SUMMARY  RECOMMENDATIONS 


The  recommendations  In  this  report  are  directed  at  the  performance 
evaluation  of  the  human  operator  during  field  test  and  evaluation. 

The  evaluation  procedures  dealt  with  here  do  not  include  the  large 
amount  of  work  carried  out  in  evaluating  the  environmental  variables 
and  their  effect  upon  the  human  operator. 

The  recommendations  are  based  upon  intensive  study  of  a  particular 
rather  complex  system  being  taken  into  the  Navy  inventory,  the  P-3C  ASW 
system,  and  upon  a  less  exhaustive  study  of  the  A-7A.  They  are  based 
also  upon  first  hand  observations  of  the  field  evaluation  process  as 
it  existed  for  these  systems  -  again  with  emphasis  upon  the  P-3C.  The 
limitations  upon  the  recommendations  resulting  from  these  conditions 
should  be  noted.  At  the  same  time,  in  arriving  at  these  recommendations, 
the  authors  have  applied  the  results  of  their  experience  in  working 
with  the  design,  development  and  evaluation  of  other  military  systems 
as  well  as  upon  the  published  findings  and  informal  communications  of 
others  with  experience  with  the  problem. 

6.1  MANAGEMENT  AND  PERSONNEL 

The  first  and  perhaps  most  important  recommendation  to  be  made 
would  help  greatly  in  alleviating  the  present  constraints  on  effective 
human  factors  field  evaluation.  This  is  that  serious  consideration  be 
given  to  the  recommendations  contained  in  Section  4.0  of  this  report. 

In  that  Section  the  need  for  assignment  to  the  program  office  of 
personnel  with  authority  and  means  for  accomplishing  human  factors 
requirements  is  stated,  recommendations  for  other  personnel  assignments 
during  system  development  and  test  are  made  and  training  and  cross 
training  suggestions  are  outlined.  The  prime  initial  requirement 
for  successful  human  factors  field  evaluation  is  the  assignment  at 
the  program  direction  level  of  a  aualified  manager  of  human  factors 
design  and  evaluation  with  authority  and  budget  adequate  to  the 
task. 


6.2  IDENTIFICATION  OF  MEASUREMENT  POINTS 

It  is  important  that  a  detailed  llstinq  of  the  operator  tasks 
and  an  Operational  Sequence  Diagram  be  generated  and  kept  current 
throughout  the  development  and  test  of  the  system.  Both  manual 
performance  and  tactical  decision  measurement  points  should  be 
identified  early  in  the  development  phase  along  with  scales  of 
measurement  and  criteria  for  acceptable  performance.  This  infor¬ 
mation  is  then  available  In  planning  tne  field  test  and  evaluation. 

The  evaluator  will  usually  find  it  necessary  to  modify  the  task 
listing  and  OSD  to  reflect  changes  in  tne  system  just  prior  to  and 
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during  tne  +©■=+  and  evaluation  Drocess.  It  is  recommended  that  he 
use  the  OSD  to  obtain,  from  either  experienced  operators  of  the 
system  or  Human  Factors  engineers  famliiar  with  the  system, 
identi f ications  of  the  points  of  human  performance  measurement, 
scales  of  measurement  and  criterion  limits.  This  procedure  is 
discussed  in  Section  5.4.7  of  the  report. 

6.3  TESTING  CONDITIONS 

The  Human  Factors  evaluator  must  be  oriented  toward  obtaining 
reliable  measures  under  test  conditions  which  will  allow  him  to 
forecast  how  the  operator  will  perform  with  the  complete  system  in 
the  operational  theater.  The  requirement  for  obtaining  reliable 
measures  and  that  of  testing  under  the  most  representative  conditions 
often  may  conflict.  When  test  conditions  are  most  like  those  of  the 
operational  theater  the  less  likely  it  will  be  that  conditions  of 
test  can  be  held  constant  from  run  to  run  and  that  reliable  measures 
can  be  obtained. 

In  striking  a  balance  between  representative  test  conditions 
and  reliability  of  data  the  field  evaluator  must  strive  to  attain 
the  most  representative  conditions  possible  while  maintaining  good 
test  procedure.  Since  similarity  to  operational  conditions  is  most 
nearly  attained  at  the  field  test  stage,  test  runs  designed  solely 
for  control  and  measurement  of  human  performance  variables  are 
necessary  and  must  be  scheduled.  At  the  same  time  the  use  of  the 
dynamic  simulator,  test  ranges  and  laboratory- 1  ike  settings  should 
not  be  ruled  out  as  a  part  of  the  field  test  procedure.  The  field 
evaluator  who  Is  familiar  with  the  system  and  tne  field  test 
constraints  and  who  has  available  experienced  operators  to  perform 
the  tasks  often  can  use  these  test  conditions  to  advantage.  This 
is  particularly  true  when  safety  considerations  or  lack  of  control 
over  test  conditions  limits  the  utility  of  the  full  system  as  a 
test  situation.  This  subject  is  discussed  in  Section  5.7. 

6.4  CRITERIA 

The  field  evaluator’s  task  is  to  obtain  reliable  performance 
data  which  he  can  compare  to  the  criterion  requirements  for 
successful  performance  and  reach  an  evaluative  judament  as  to 
the  adequacy  of  deslan.  Measurements  must  be  taken  which  al low 
a  direct  comparison  of  performance  with  the  level  of  performance 
requ i red. 

When  specific  criteria  are  not  available  the  evaluator  must 
obtain  actual  performance  data  over  a  series  of  runs  upon  which 
to  base  his  predictions  of  whether  the  performance  will  s I qn i f i cant  I y 
affect  the  successful  performance  of  the  mission  by  the  tctal  svstem. 
The  sampling  conditions  under  which  he  collects  these  predictive  data 
are  therefore  critical  to  his  predictions.  The  problem  of  performance 
criteria  is  discussed  more  fully  In  Section  5.6.3. 
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o.5  OPERATOR  WORKLOAD 
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The  evaluation  of  human  operator  performance  may  be  simply  stated 
as  the  assessment  of  the  accuracy  and  timeliness  with  which  he  (I) 
processes  data  at  the  man-equipment  interface  and  (2)  makes  the 
tactical  decisions  necessary  for  the  system  to  accomplish  its  mission. 
Althouqh  assessments  of  workload  will  have  been  made  earlier  durinq 
the  development  of  the  system,  the  determination  of  whether  the  operator 
can  successfully  accomplish  a  sequence  of  tasks  in  the  time  available 
for  accomp I i sh i nq  it  within  the  actual  mission  must  be  determined 
under  conditions  as  nearly  representative  of  the  operational  mission 
as  possiDle.  Tendencies  to  qet  behind  the  task  requirements  or  to 
substitute  other  procedures  should  be  invest iqated  for  possible 
design  inadequacies  under  the  assumption  tnat  the  stress  of  actual 
operational  theater  operations  will  tend  to  worsen  operator  performance. 

6.6  PERSISTENCE  OF  FORMER  PROCEDURES  AND  HABITS 

It  will  be  particularly  important  to  the  Human  Factors  evaluator 
to  set  up  observational  and/or  report  procedures  for  determininq  the 
degree  to  which  deviations  from  design  procedures  occur  through  use 
of  earlier  learned  habits.  These  pose  a  particular  problem  because 
of  the  possibility  of  the  reversion  to  these  procedures  and  habits 
during  critical  and  stressful  situations.  This  will  be  more  likely 
to  occur  when  the  Information  displays  are  essentially  the  same  as 
formerly  used  and  the  responses  to  the  information  (actions  and 
procedures)  have  been  changed. 
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Note : 
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SUGGESTED  OUTLINE  FOR  COURSE  IN 
HUMAN  FACTORS  TEST  AND  EVALUATION 


Material  suggested  in  this  outline  comes  in  part  from 
Smode,  et  al.,  1962.  The  use  of  that  report  Is 
suggested  as  a  suitable  part  of  the  text  material  for 
the  course  outlined  in  this  Appendix. 
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APPENDIX  A 


This  appendix  outlines  a  course  of  study  designed  for  field 
test  personnel  whose  particular  Interest  and  concern  Is  human 
factors  testing.  The  course  Is  Intended  to  cover  Items  essential 
to  an  overall  orientation  In  human  factors  test  and  evaluation 
and  to  provide  methods  and  techniques  necessary  to  effective 
eva I uat i on . 

In  the  outline  a  differentiation  is  made  between  field 
research  and  field  evaluation  with  similarities  and  differences 
drawn  between  them.  While  the  primary  atm  of  the  course  Is  to 
provide  guidance  for  field  evaluation,  points  relevant  to  field 
research  are  given  In  order  that  the  human  factors  evaluator  may 
have  some  guidance  In  field  research  techniques  should  the 
opportunity  arise  to  apply  them  during  a  field  evaluation. 


COURSE  OUTLINE 

1.0  Distinction  between  field  research  and  field  evaluation. 

1 . 1  Field  Research 

1.1.1  More  opportunity  to  identify  and  control  the 
parameters  and  variables  of  Interest. 

1.1.2  Usually  collecting  data  to  establish  norms  or 
make  comparisons. 

1.1.3  More  opportunity  to  introduce  special 
Instrumentation  to  obtain  performance  measures. 

1.1.4  More  flexibility  in  changing  procedures  and 
equipment  as  testing  progresses  to  achieve  the 
desired  goals  of  testing. 

1.1.5  May  be  testing  to  determine  how  well  system, 
subsystem  or  component  meets  some  set  or 
required  level  of  performance. 

1.2  Field  Evaluation 

1.2.1  Nearly  always  testing  to  determine  whether  system 
performance  meets  some  specified  level  of  design 
performance. 


1.2.2  Nearly  always  there  is  a  limitation  on  time  within 
which  the  evaluation  is  to  be  performed. 

1.2.3  Often  must  fit  measures  of  man's  performance  into 
the  tests  of  equipment. 

1.2.4  Little  or  no  opportunity  to  vary  independent 
variables  systematically. 

2.0  General  setting  within  which  field  research  and  evaluation  are 
conducted. 

2.1  Some  fixed  time  span. 

2.2  Testing  both  equipment  and  men. 

2.2.1  Some  tests  peculiar  to  equipment  alone,  e.a.,  how 
it  functions  under  the  field  conditions. 

2.2.2  Some  tests  peculiar  to  man  alone,  e.g.,  his 
physiological  state  under  the  field  conditions. 

2.2.3  Some  tests  peculiar  to  the  interaction  between  men 
and  equipment,  e.q.,  how  well  man  can  operate  or 
maintain  eauipment  under  the  field  conditions. 

2.3  Human  factors  testing  often  goes  "piggy  back"  on  equipment 
testinq,  I.e.,  must  be  fitted  in  and  around  equipment 
testing.  This  is  a  fact  of  human  factors  testinq  which 
must  be  recognized  in  sett i no  up  human  factors  research 

and  evaluations  in  the  field.  A  qreat  deal  of  human  factors 
data  collecting  can  be  carried  out  in  conjunction  and 
simultaneous  with  equipment  testing.  However,  It  will 
be  necessary  to  program  specific  blocks  of  time  for 
collectinq  human  factors  data  independent  of  equipment 
tests. 

3.0  Requirement  for  thorough  knowledge  of  the  system  under  test, 
its  operation  and  its  operating  environment. 

3.1  To  isolate  and  define  the  important  parameters  which 
may  influence  performance. 

3.2  To  set  up  methods  of  either  controlling  or  systematically 
varying  these  parameters. 

3.3  To  determine  what  measures  are  appropriate  and  at  what 
test  points  they  will  be  taken. 
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4.0  Metnods  for  determining  necessary  system  detai Is. 

4.1  Data  flow  analysis. 

4.1.1  What  inputs  are  (or  must  be)  received  by  each 
component  or  subsystem  and  what  outputs  are 
(or  must  be)  made  to  the  next  component(s)  or 
suosystem(s) . 

4.1.2  For  man  as  an  information  processor  we  are 
interested  in  what  information  he  must  receive 
or  is  the  system  designed  for  him  to  receive, 
what  transformations  of  the  information  he  must 
make,  and  what  "informational"  outputs  he  must 
make  to  other  components  of  the  system. 

4.1.3  A  number  of  formats  with  their  own  nomenclature 
or  symbology  have  been  developed.  Each  has  its 
own  particular  assets. 

4.2  Time  Line  Analysis  -  serves  both  as  a  means  of  learning  the 
system  and  evaluating  workload. 

4.3  Operational  Sequence  Diagram  -  serves  as  means  of  learning 
the  system  and  identification  of  manual  performance  and 
tactical  decision  measurement  points. 

5.0  Some  general  principles  of  sound  research  and  evaluation  procedure 

for  whicn  to  aim. 

5.1  Standardization. 

5.1.1  Test  conditions, 

5.1.2  Performance  measures. 

5.1.3  Observers. 

5.1.4  Environmental  effects. 

5.2  Control  or  assessment  of  relevant  variables. 

5.2.1  If  variable  cannot  be  controlled  it  should  be 
measured  at  the  time  dependent  variables  are 
measured. 

5.3  Explicit  statement  of  independent  and  dependent  variables 
wilh  no  variation  in  their  meaning  or  method  of  measurement. 


5.4  Dependent  variables 


5.4.1  These  are,  in  effect,  precise  statements  of  what 
we  want  to  know  about  how  the  system  or  the  man 
within  the  system  performs. 

5.4.2  Must  order  dependent  variables  in  order  of 
importance  since  resources  and/or  time  are 
usual ly  I imited. 

5.4.3  Must  decide  upon  points  in  system  at  which 
measurements  will  be  taken. 

5.4.4  Must  determine  the  method  of  measurement,  i.e., 
direct  recording,  direct  observation,  rating 
scales,  guestionnai re,  etc. 

5.4.5  Always  detail  in  advance  the  method  of  data  reduction, 
analysis  and  presentation.  Shotgun  approach  to  data 
collection  is  not  feasible  in  field  situations. 

5.5  General  classes  of  dependent  variables. 

5.5.1  Man's  outputs. 

5. 5.1. 1  Time  to  perform. 

5.5. 1.2  Accuracy  of  performance. 

5.5.2  Man's  inputs. 

5.5.2. I  Control  displacements  or  forces. 

5.5.3  Man's  physiological  state. 

5.6  Independent  Variables. 

5.6.1  In  field  research  may  have  the  opportunity  to 
assess,  control  or  vary  systematically  the 
independent  variables.  In  field  evaluation 
usually  have  opportunity  only  to  assess  these 
vari ab les. 

5.6.2  From  system  and  task  analysis  identify  and 
describe  both  system  and  environmental  parameters 
likely  to  affect  performance. 

5.6.3  Define  and  specify  how  independent  variable  is 
measured  and  over  what  range  it  will  be  varied. 
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6.0  Criteria 


6.1  Implies  some  value  judgment  as  to  the  "goodness"  of  the 
performance. 

6.2  Measurement  per  se  does  not  provide  value  judgments. 

6.3  These  value  judgments  must  be  expressed  in  terms  of  the 
defined  purpose  or  mission  of  the  system. 

6.4  Ultimate  vs.  actual  criteria. 

6.4.1  Seldom  possible  to  obtain  direct  measures  of  tne 
ultimate  criteria. 

6.4.2  Usually  necessary  to  select  some  actual  (intermediate) 
crlter ia. 

6.4.3  Must  then  use  these  actual  (intermediate)  criteria 
in  evaluating  performance. 

6.4.4  There  is  no  certain  method  for  specifyino  the  actual 
cr i teri a. 

6.4.5  Sources  of  error  in  selecting  actual  criteria. 

6.4.5. 1  Unre  M  abi 1 1 ty . 

6. 4. 5. 2  Irrelevancy  -  the  lack  of  relation  to 
ultimate  criterion. 

6.4.5. 3  Contamination  -  inqredients  in  the  actual 
criteria  which  do  not,  in  fact,  exist  in 
the  ultimate  criterion. 

6. 4. 5. 4  Distortion  -  errors  arisina  from  assign'na 
incorrect  weights  to  the  separate  factors 
that  comprise  the  actual  criteria. 

6.5  Establishino  valid  criteria. 

6.5.1  No  established  procedures. 

6.5.2  Recoqnizing  importance  of  criteria  selection  and 
types  of  errors  which  might  be  present  are  nood 
starting  points. 
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6.5.3  Steps  which  should  lead  to  more  useful  and  relevant 
criteria. 

6.5.3. 1  Define  the  actlviiy  -  specify  to  extent 
possible  the  activity  desired  for 
successful  and  proficient  performance. 

6. 5. 3. 2  Analyze  the  activity  -  consider  the  activity 
in  terms  of  purposes  or  goals,  behavior  and 
skills  involved,  their  relative  Importance 
and  standards  of  performance  expected. 

6. 5. 3. 3  Define  successful  performance. 

6. 5. 3. 4  Develop  sub-criteria  to  measure  each  element 
of  success. 

6. 5. 3. 5  As  appropriate  develop  a  combined  measure  of 
successful  performance. 

6.6  Combining  criteria. 

6.6.1  Often  necessary  that  several  criteria,  all  of  which  are 
relevant  for  a  particular  activity  be  used,  in  such 
case  It  may  be  desirable  to  combine  them  into  a  single 
comp  rehens i ve  one . 

6.6.2  Combining  will  usually  involve  assigning  relative 
weights  to  the  individual  criteria. 

6.6.3  Rules  for  combining  criteria. 

6.6.3. 1  Weight  in  accordance  with  their  relevance 
to  the  ultimate  criterion. 

6.6. 3.2  Criteria  which  repeat  or  overlap  factors 

in  other  criteria  should  receive  low  weight. 

6.6. 3. 3  Other  things  beina  equal  the  more  reliable 
criteria  should  receive  more  weight. 

6.6.4  Caution  must  be  exercised  in  applyina  weights  to 
raw  score  values  -  use  standard  scores. 

7.0  Measurement  of  performance. 

7. I  What  to  measure. 

7.1.1  Should  be  preceded  by  an  explicit  statement  of  the 
research  questions  being  asked. 


7.1.2  Need  analysis  of  the  system  which  allows  Identification 
of  the  points  within  the  system  at  which  performance 
can  be  measured  and  recorded. 

7.1.3  Identification  of  critical  tasks. 

7. 1.3. 1  Usually  not  feasible  to  measure  all  points 
Identified  so  must  select  for  measurement 
those  tasks  on  which  good  performance  leads 
to  mission  success  and  poor  performance 
leads  to  mission  failure. 

7. 1.3.2  In  identifying  critical  tasks  asking  the 
following  questions  with  respect  to  the 
tasks  is  helpful. 

Would  be  low-minimum  performance; 

.  lead  to  an  accident? 

.  result  directly  in  mission  failure? 

.  be  impossible  to  remedy  within  the  time 
constraints  or  not  at  all? 

.  be  difficult  to  detect  because  of 
inadequate  information  feedback? 

.  recur  over  time  In  such  a  way  as  to 
produce  a  cumulative  effect? 

.  contribute  a  large  proportion  of  time 
to  the  total  time  required  for  some 
larqer  and  critical  function? 

7.2  Levels  of  measurement 

7.2.1  Nominal  scale. 

7.2.2  Ordinal  scale. 

7.2.3  Interval  scale. 

7.2.4  Ratio  scale. 

7.3  Specificity  of  measures. 

7.3.1  Over-all  measures. 
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7.3.1. 1  Global  Indices  of  sub-system  or  system  A 

performance  associated  with  mission  j 

segments  or  complete  mission. 

7. 3. 1.2  Useful  in  assessment  since  it  is  descriptive 
of  some  end  result  which  can  be  compared 
with  the  standard. 

7. 3. 1.3  Weak  in  analytic  sense  since  they  provide 
no  detailed  information  on  performance 
beyond  the  outputs  sampled. 

7.3.2  Diagnostic  measures. 

7.3.2. 1  Quite  specific,  identifying  elements  of 
job  performance  in  specific  skill  areas. 

7. 3.2.2  Since  they  are  concerned  with  smaller  more 
precisely  defined  units  of  behavior  they 
lend  themselves  more  readily  to  objective 
measurement. 

7.4  Accuracy  of  measurement, 

7.4.1  Refers  to  how  close  the  obtained  value  or  measure 

is  to  the  true  value. 

7.4.2  There  is  no  single  way  to  assure  measurement  accuracy. 

Accuracy  may  be  improved  by  the  following  means. 

7.4.2. 1  Increase  scope  of  measurement  to  be  taken  - 
incluoe  additional  aspects  of  relevant 
behavior. 

7. 4. 2. 2  Increase  the  number  of  observations  on  which 
summary  statistics,  e.g.,  are  based. 

7.4.2. 3  Control  the  conditions  under  which  measurements 
are  taken. 

7.5  Reliability  of  measurement. 

7.5.1  uefinition  -  agreement  or  consistency  of  measures  from 

repeated  observations. 

7.5.2  Relation  between  reliability  and  validity. 

7.5.3  Absolute  expression  of  reliability  -  standard  error  of 

measurement. 

7.5.4  Relative  measures  of  reliability  -  expressed  in  terms 

of  correlation. 
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7.6  Validity  of  measurement 


7.6.1  Definition  -  deqree  to  which  measuring  instruments 

measure  what  they  are  intended  to  measure. 

7.6.2  Four  types  of  validity. 

7.6.2. 1  Content  validity  -  loqical  validity  based  on 
expert  oDinion  or  oTher  loqical  considerations. 

7. 6. 2. 2  Concurrent  validity  -  statistical  validity  - 
correlation  with  other  task  or  dimension 
external  to  the  measurement. 

7.6.2. 3  Predictive  validity-statistical  correlation 
between  obtained  measures  and  future  states 
on  some  task  or  dimension  external  to  the 
measurement. 

7. 6. 2. 4  Construct  validity  -  loqical  validity  -  where 
the  emphasis  is  on  the  trait,  quality  or  aoiliTy 
presumed  to  underlie  the  measures  being  taken. 

7.7  Objective  (quantitative)  vs.  subjective  (qualitative)  measures. 

7.7.1  Objective  measures. 

7.7.1. 1  Generally  permit  measurement  relatively 
independent  of  the  observer, 

7.7. 1.2  Generally  of  higher  reliability  than  subjective. 

7.7. 1.3  Greatest  objectivity  obtained  by  means  of 
recording  instruments  where  a  permanent  record 

of  behavior  is  obtained  at  the  time  of  occurrence. 

7.7. 1.4  Insistence  upon  complete  objectivity  tends  to 
result  in  omission  of  a  variety  of  critical 
job  components  because  of  inability  to  measure 
them  objectively, 

7.7. 1.5  Can  result  in  impractical  gadgetry  and  procedures. 

7. 7. 1.6  Relatively  free  from  observer  bias. 

7.7.2  Subjective  measures. 


7.7.2. 1  Generally  dependent  upon  the  characteristics 

of  the  observers  -  may  introduce  bias. 

7. 7.2.2  Inter-observer  reliability  not  always  hiah. 

7. 7.2. 3  More  flexibility  in  administration. 

7.7.3  Ratinqs  (a  form  of  subjective  measurement). 

7.7.3. 1  Ratinn  procedures. 

.  Ratinq  scales  -  rater  makes  judoment  on 
scale  of  defined  cateoories. 

,  Comparative  systems  -  pair  people  or  units 
with  respect  to  each  other. 

.  Check  lists  -  judqments  bv  raters  as  to 
which  of  a  series  of  descriptive  terms 
either  are  or  are  not  applicable  to  the 
units  being  evaluated. 

.  Critical  incidents  -  recording  actual 

incidents  as  behaviors  which  are  especial Iv 
effective  or  ineffective  in  the  accomplish¬ 
ment  of  the  mission  -  Standina  of  a  unit  is 
indicated  by  frequency  of  occurrence  of 
reported  incidents. 

7. 7. 3. 2  The  sources  of  bias  in  ratines. 

.  Halo  effect. 

.  Leniency  error. 

.  Error  of  central  tendency. 

.  Contrast  error  -  tendency  to  r  it  :  in 
opposite  direction  on  a  dimension  fror 
how  the  raters  see  themselves. 

.  Proximitv  error  -  tendency  for  ratinas 
to  be  more  related  when  made  close  to 
each  other  in  time. 

7.8  Individual  vs.  crew  performance. 

7.8.1  Crew  performance  musr  be  regarded  as  more  than  the 

sum  of  the  individual  performances. 
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7.8.2  Measures  of  crew  performance. 

7.8.2. 1  Synchronization  of  action. 

7. 8. 2. 2  Response  Improvisation. 

7.8.2. 3  Amount  of  time  spent  interaction  -  aood 
crew  should  reduce  individual  interaction 
to  a  minimum  so  th*+  more  effort  ;s 
devoted  to  the  job  and  less  to  coordination, 

7. 8. 2. 4  Amount  of  commun i cat  ion  -  the  less  the 
communication  the  higher  the  degree  of 
coord i nat I  on. 

7. 8. 2. 5  Freedom  for  i nterpersona I  commun i cat i ons. 

7. 8. 2. 6  Monitoring  and/or  making  some  responses 
for  another  crew  member. 

7.8.2. 7  Aiding  in  the  detection  of  out-of-tolerance 
conditions. 

7. 8. 2. 8  Sharing  of  risk  activities  among  crew 
members.  . 

8.0  Procedural  steps  in  assessment  of  performance. 

8.1  Conduct  thorough  analysis  of  the  tasks  and  generate  Operational 
Sequence  Diagram  or  Mission  Time  Line. 

8.2  Select  test  points  and  measures  appropriate  to  the  behavior 
to  be  evaluated. 

8.3  Define  performance  reauirements  of  tnc  task  as  appropni ate. 

8.4  Identify  important  and  critical  aspects  of  the  task  and 
the  envi ronment, 

I 

8.5  Determine  conditions  under  which  measures  will  be  taken. 

8.6  Determine  techniques  for  obtaining  measurement  data  and  for 
combining  measures  as  appropriate. 

8.7  Specify  methods  of  data  analysis. 


A- 1 2 


9.0  Subjects  or  operators  upon  which  data  is  collected  (subject 
samp  ling) 

9. 1  Sample  size. 

9.2  Sample  composition  with  respect  to  experience  and  other 
factors  and  its  relation  to  purposes  of  measurement  and 
usefulness  of  the  data. 

10.0  Data  col  lection  and  treatment. 

10.  I  Experimental  methods. 

10. 1. 1  Single  variable  design. 

10.1.2  Multl-varlate  design. 

10. 1.2. 1  Each  subject  his  own  control. 

10.1.2.2  Independent  qroups. 

10.2  Specific  measures  of  performance. 

10.2.1  Procedural  tasks. 

1 0 . 2 . 1 . 1  Time. 

10.2.1.2  Accuracy. 

10.2.2  Closed-loop  trackinn  accuracy  (compensatory  and  Dursuit) 

10.2.2.1  Integration  of  error 

10.2.2.2  Number  of  crossings. 

10.2.2.3  Time  on  tarqet. 

10.2.2.4  Frequency  of  catastrophic  errors. 

10.2.3  Operator  output. 

10.2.3.1  Power  density  spectrum. 

10.2.3.2  Standard  deviation  about  mean  output. 

10.2.4  Crew  coordination. 

10.2.4.1  Overall  gross  measure  of  crew  output  in  terms 
of  time  to  accomplish  task  and  accuracy  in 
accomp I i sh i ng. 
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10.2.4.2  Number  of  communications  between  members. 


10.2.5  Decision  making  tasks. 

10.2.5.1  Time. 

10.2.5.2  Accuracy. 

10.2.6  Perceptual  and  motor  skills. 

10.2.6.1  Psychopnysical  measures. 

10.3  Parametric  statistics. 

10.3.1  Measures  of  central  tendency. 

10.3.2  Measures  of  deviation  from  standard  (CE). 

10.3.3  Measures  of  variability. 

10.3.4  Measures  of  correlation. 

10.3.5  Tests  of  reliability  of  differences. 

10.4  Non-parametri c  statistics. 

10.4.1  Difference  from  parametric  statistics. 

10.4.2  Tests  of  reliability  of  differences. 

10.4.3  Test  of  correlation. 

10.5  Presentation  of  results. 

10.5.1  Pictorial  graphs  or  charts. 

10.5.2  Significance  tables  -  differences  in  terms  of 
likelihood  function. 
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APPENDIX  B 


SAMPLE  OPERATIONAL  SEQUENCE  DIAGRAM 
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CONSTRUCTION  OF  BEHAVIORAL  CODE  WORDS 


FIRST  LETTER 


HUMAN  OR  MACHINE 

H 

(Human ) 

M 

(Mach i ne ) 

SECONu  LETTER 

oEHAV 1  OR 

A 

(Act ) 

D 

(Decide) 

T 

(Transmi t  > 

If 

( R.ece  i  ve ) 

S 

( Store) 

P 

(Use  Previously  Stored  Information) 

M 

(Mon i tor ) 

THIRD  LETTER 

MEANS  OF  PERFORMANCE 

S 

( Soc-ech ) 

F 

(Pnonc,  Sound-power) 

1 

(Tntercom) 

E 

( E 1 ectr i ca 1 

or  Electronic) 

T 

(Touch ) 

V 

( V i sua  1  ) 

r 

( F i led) 

FOURTH  LETTER 

DISPLAYED  UR  NOT 

u 

( Di sp 1 ayed ) 

Bl  anK 

(Not  Disp 

1  ayed ) 

FIFTH  LETTER 

INVERSE  MEANING  * 

G 

(Co,  ves,  normal,  etc.) 

N 

(No-Go,  no, 

abnorma 1 ,  etc . ) 

*  Although  this  symbol  is  not  used  in  the  illustrations  to  follow  it 
may  be  used  where  necessary  or  for  convenience  to  indicate  the 
opposite  or  lack  of  an  action,  i.e,,  no-qo.  (See  Wilson,  D.  A.,  I9bb). 
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OPERATIONAL  SEQUENCE  DIAGRAM 


HRV 


_LL 


HRV 


fir 


TACCO  COMPUTER 

(MRE>- 

1.  Receive  Initial  Contact  Report  (M-14) 

2.  Acknowledge  Contact  Report 

•-(MTE)  — 

3.  Request  Classification  and  Bearing 

Line 


.-(MTE) 


—  <  MRE  b- 


4.  Receive  Acknowledgement  of  Request 


5.  Rescale  MOD  as  Required 


•••(MTE)  --(MAE)  Display  Desired  Scale 


(MRE)- 


6.  Observe  MDO  Rescaled 


7.  Observe  Location  of  M-14 


(MRE)-* 


(MRF>— * 

Assimilate  NAV  and  Buoy  RF  Data 
and  Display  Location  of  M-14 


(MRE)—. 


8.  Observe  Navigation  Data 

-  (MRE)— 

9.  Observe  Position  of  Aircraft  Symbol 

(MRE)— 

10.  Observe  Direction  that  Aircraft 

Symbo 1  i s  Mov i ng 


T 

(J^Ed) 


(MRF>— 


Assimilate  NAV  Data  and  Display 
Position  of  Aircraft  Symbol 

-  tMRF)— 


Assimilate  NAV  and  Time  Data 
and  Display  Change  of  Aircraft 
Symbol  Position 


II.  Request  Pilot  Turn  Toward  M-14 

>-(MTE) 


(MRE  b* 


12.  Receive  Acknowledgement  of  Request 

(MRE)—  • 


(MPED) 


13,  Observe  Aircraft  Symbol  Turning 


(MRF>- 

Assimilate  NAV  and  Time  Data  and 
Display  Aircraft  Symbol  Turning 


(MRE)— 


14.  Receive  Classification  Report 


T 
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(SYSTEM 

P-3C  ASW  AIRCRAFT 

PAGE  1 

OF  10 

lOPERATOR  - 

TACCO 

REVISION 

NO  0 

^ PHASE 

SEARCH 

DATE  14 

JUN  1971 

EVENT 

INITIAL  CONTACT 

i 

T  A  S 

K  SEQUENCE 

RELATED  COMPUTER  OPERATION 

BEHAVIOR  i 

BEHAV 1  OR 

CODES 

CODES 

‘MAED  DISPLAY  CONTACT  SYMBOL 


t  ^MRE+ 

OBSERVE  CONTACT  SYMBOL  HRV^ 

i  I  ; 

PRESS  ACK  CONTACT  1  HAT\ 


SWITCH 


OBSERVE  CONTACT  SYMBOL  HRV-^ 
REMOVED  :  ! 


OBSERVE  POSITION  OF  HRV^ 
AIRCRAFT  RELATIVE  TO 
CONTACT  BUOY  ! 


>1AE  i  REMOVE  CONTACT  SYMBOL 


MTE-^' 


IRF— MPED  DISPLAY  AIRCRAFT  AND 
xMTE-*'"  CONTACT  BUOY  SYMBOLS 


OBSERVE  DIRECTION  OF  HRVy 
AIRCRAFT  MOVEMENT  |  | 


MPED  DISPLAY  AIRCRAFT  SYMBOL 
MTE-*  MOVEMENT 


REQUEST  PILOT  TO  BEGIN  i  HTI 
TURN  TO  STARBOARD  I  | 


MOVE  HOOK  SYMBOL  TO  FTP  HAT 


^MTE-*- 


El 

;  I  ^MTE  v 

1  ^RE  ^ 

^MAED 

>  1  j/MTE'*' 

t  'MRE^** 

OBSERVE  HOOK  SYMBOL  AT 

HRV  * 

FTP  El 

1 

PRESS  HOOK  VERIFY 

HAT\ 

%TEv  . 

^MAED 

;  |  :  -MTE^" 

T  MRE^ 

SW 1 TCH 

QBSERVE  FLASHING  SYMBOL  HRV> 

PRESS  DESTROY  PT  DATA 

? 

HAT  x 

i  .  ...  .  . 

MAEQ  DISPLAY  FLASHING  SYMBOL 
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APPENDIX  C 


ILLUSTRATION  OF  OPERATIONAL  SEQUENCE  DIAGRAM 
AND  INSTRUCTIONS  FOR  IDENTIFYING  MEASUREMENT 
POINTS,  TYPE  OF  DATA  WHICH  MAY  BE  COLLECTED 
AND  CRITERION  LEVEL  OF  ACCEPTANCE  PERFORMANCE 
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from  the  system,  (<:)  interpret  the  information,  and  (3)  put  information  into  the  system. 


one  task  or  taSK  sequence  is  selected  on  one  paqe,  use  a  correspond i nq  number  of  asterisks. 


task  where  the  operator  makes  an  input  to  the  system? 


difference  between  acceDtable  and  unacceptable. 
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